How to Use Google Custom Search Engine API in Python

Learning how to create your own Google Custom Search Engine and use its Application Programming Interface (API) in Python.
Abdou Rockikz · 6 min read · Updated mar 2020 · General Python Topics


Google Custom Search Engine (CSE) is a search engine that is suited for developers in which it lets you include a search engine in your application, whether it is a website, a mobile app or anything else.

Since manually scraping Google Search is highly unsuggested, as it will restrict with a reCAPTCHA every few queries, in this tutorial, you will learn how you can setup a CSE and use its API in Python.

In CSE, you can customize your engine that searches for results on specific websites, or you can use your website only. However, for this tutorial, we will enable our search engine to search the entire web.

Setting Up a CSE

First, to setup your search engine, you need to have a Google account. After that, head to CSE page and sign in to Custom Search Engine as shown in the following figure:

Sign In to Custom Search Engine

After you login to your Google account, a new panel will appear to you that looks something like this:

Creating a new CSE

You can include the websites you want to include your search results, choose a language of your search engine, and set up the name of it. Once finished, you'll be redirected to this page:

CSE Created

Using CSE API in Python

Now to use your Search Engine in Python, you need two things: First, you need to get your Search Engine ID, you can get easily find it in CSE control panel:

Search Engine ID in Control Panel

Second, you have to generate a new API key, head to Custom Search JSON API page and click on "Get a Key" button there, a new window will appear, you need to create a new Project (you can name it whatever you want) and click on Next button, after that you'll have your API key, here is my result:

CSE API Key is Ready to use

Lastly, if you want to search the entire web, you need to enable it on your control panel:

Turning on Search the Entire Web in CSENow you have everything you need to use the CSE API in your Python code, open up a new Python file and follow along, we'll be using requests library for convenience, you can install it using this command:

pip3 install requests

Let's initialize our CSE requirements:

import requests

# get the API KEY here: https://developers.google.com/custom-search/v1/overview
API_KEY = "<INSERT_YOUR_API_KEY_HERE>"
# get your Search Engine ID on your CSE control panel
SEARCH_ENGINE_ID = "<INSERT_YOUR_SEARCH_ENGINE_ID_HERE>"

For demonstration purposes, we will do a simple search query, "python". Let's build the API URL we'll request:

# the search query you want
query = "python"
# constructing the URL
# doc: https://developers.google.com/custom-search/v1/using_rest
url = f"https://www.googleapis.com/customsearch/v1?key={API_KEY}&cx={SEARCH_ENGINE_ID}&q={query}"

Making the API request and using requests' json() method to automatically parse the returned JSON data to a Python dictionary:

# make the API request
data = requests.get(url).json()

Now this data is a dictionary that contains a lot of result tags, we are only interested in "items", which are the search results, by default, CSE will return 10 search results, let's iterate over them:

# get the result items
search_items = data.get("items")
# iterate over 10 results found
for i, search_item in enumerate(search_items, start=1):
    # get the page title
    title = search_item.get("title")
    # page snippet
    snippet = search_item.get("snippet")
    # alternatively, you can get the HTML snippet (bolded keywords)
    html_snippet = search_item.get("htmlSnippet")
    # extract the page url
    link = search_item.get("link")
    # print the results
    print("="*10, f"Result #{i}", "="*10)
    print("Title:", title)
    print("Description:", snippet)
    print("URL:", link, "\n")

We are only extracting the title, the snippet (i.e description) and the link of resulting page, check the result:

========== Result #1 ==========
Title: Welcome to Python.org
Description: The official home of the Python Programming Language.
URL: https://www.python.org/

========== Result #2 ==========
Title: The Python Tutorial — Python 3.8.2 documentation
Description: It has efficient high-level data structures and a simple but effective approach to 
object-oriented programming. Python's elegant syntax and dynamic typing,
together ...
URL: https://docs.python.org/3/tutorial/

========== Result #3 ==========
Title: Download Python | Python.org
Description: Looking for Python with a different OS? Python for Windows, Linux/UNIX, Mac OS     
X, Other. Want to help test development versions of Python? Prereleases ...
URL: https://www.python.org/downloads/

That's awesome, I have snipped the output to only 3 search results, but it will print 10 results in your case.

Conclusion

You can specify various request parameters to customize your search, you can also print the data dictionary to see other meta data, such as total results, search time, and even the meta tags of each page. Check CSE's documentation for further detail.

By default, for the free access, it is limited to 100 search queries per day, if you feel that isn't enough for your application, then consider signing up for billing in the API Console.

Now you can integrate searching techniques on your applications, I hope this tutorial was helpful and easy to follow. If you like it, please don't hesitate to share it !

Read also: How to Download Files in Python.

Happy Coding ♥

View Full Code
Sharing is caring!



Read Also





Comment panel

   
Comment system is still in Beta, if you find any bug, please consider contacting us here.