pythonpython-2.7google-search

Searching in Google with Python


I want to search a text in Google using a python script and return the name, description and URL for each result. I'm currently using this code:

from google import search

ip=raw_input("What would you like to search for? ")

for url in search(ip, stop=20):
     print(url)

This returns only the URL's. How can I return the name and description for each URL?


Solution

  • Not exactly what I was looking for, but I found myself a nice solution for now (I might edit this if I will able to make this better). I combined searching in Google like I did (returning only URL) and the Beautiful Soup package for parsing HTML pages:

    from googlesearch import search
    import urllib
    from bs4 import BeautifulSoup
    
    def google_scrape(url):
        thepage = urllib.urlopen(url)
        soup = BeautifulSoup(thepage, "html.parser")
        return soup.title.text
    
    i = 1
    query = 'search this'
    for url in search(query, stop=10):
        a = google_scrape(url)
        print str(i) + ". " + a
        print url
        print " "
        i += 1
    

    This gives me a list of the title of pages and the link.

    And another great solutions:

    from googlesearch import search
    import requests
    
    for url in search(ip, stop=10):
                r = requests.get(url)
                title = everything_between(r.text, '<title>', '</title>')