pythoncssweb-scrapingbeautifulsouppython-webbrowser

Opening top google search results in Python

I am trying to open the top 5 search results in google. But my code is not opening the top results. Instead, it is opening 5 tabs with google, google web results, google images, google news, and google books. My code is below,

import requests, sys, webbrowser, bs4

res = requests.get('https://google.com/search?q=' + ' '.join(sys.argv[1:]))
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')
linkElems = soup.select(r'a')
numOpen = min(5, len(linkElems))
for i in range(numOpen):
    webbrowser.open('https://google.com' + linkElems[i].get('href'))

Please help. I want the code to open the top five search results and not images or books.

Solution

  • As mentioned select your elements more specific but try to avoid using dynamic class names, instead try css selectors:

    soup.select('a:has(h3)')

    Example

    import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get('https://google.com/search?q=test',headers = {'User-Agent': 'Mozilla/5.0'}, cookies={'CONSENT':'YES+'}).text)
soup.select('a:has(h3)')