pythonweb-scrapingbeautifulsouprequest

I am trying to scrape but my command prompt prints nothing. What am I doing wrong?

I am trying to print out headlines. Here is my code:

import requests
from bs4 import BeautifulSoup

base_url = 'http://www.nytimes.com'
r = requests.get(base_url)
r_html = r.text
soup = BeautifulSoup(r.text, features="lxml")

for story_heading in soup.find_all(class_="balancedHeadline"):
    if story_heading.a:
        print(story_heading.a.text.replace("\n", " ").strip())
    else:
        print(story_heading.contents[0].strip())

My command prompt gives me no response. It prints nothing. What is wrong with my code?

Solution

  • The list soup.find_all(class_="balancedHeadline") returns is empty. There seems to be no class "balancedHeadline". I didn't check the class scheme, but you'll get at least some results for class_="css-1m5bs2v esl82me0".