pythonweb-scrapingbeautifulsouprequest

How can I scrape all the courses from the website?

I want to scrape every course name on this page [this page][1] Here is my code:

URL = 'https://www.bachelorsportal.com/search/bachelors-degrees/computer-science-it/#q=di-24|lv-bachelor?'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')

results = soup.findAll(class_='StudyTitle')
print(results)

I'm getting empty array. Why?

Solution

  • The @baduker is right, the courses' content gets dynamically loaded thru JS. See the URL's main request empty response:

    enter image description here

    The obvious solution is to use a browser automation by selenium (eg. headless Chrome).

    Hack

    You might find the XHR/Ajax requests that return courses' content (as json) and perform them by Python to fetch content.