How can I scrape all the courses from the website?
I want to scrape every course name on this page [this page][1] Here is my code:
URL = 'https://www.bachelorsportal.com/search/bachelors-degrees/computer-science-it/#q=di-24|lv-bachelor?'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
results = soup.findAll(class_='StudyTitle')
print(results)
I'm getting empty array. Why?
Solution
The @baduker is right, the courses' content gets dynamically loaded thru JS. See the URL's main request empty response:
The obvious solution is to use a browser automation by selenium (eg. headless Chrome).
Hack
You might find the XHR/Ajax requests that return courses' content (as json) and perform them by Python to fetch content.
- coc-pyright fail to report reportAttributeAccessIssue if __setattr__ is provided
- PySide6 signal not emitting properly when passed to decorator
- Ruff does not autofix line-too-long violation
- How to avoid a Torch error with Open WebUI/Ollama
- How to configure header filters in a HoloViz Panel Tabulator widget?
- How to read pickle file?
- Open a webpage popup window in Microsoft Edge browser using Python without Selenium
- Forbid usage of defaults channel in Conda
- How to tackle "Statement is unreachable [unreachable]" with mypy when setting attribute value in a method?
- wave.Error: unknown format: 3 arises when trying to convert a wav file into text in Python
- Launch Pycharm from command line (terminal)
- How to write type hints for recursive function computing depth in Python?
- Type annotations: pathlib.Path vs importlib.resources.abc.Traversable
- How can I do a line break (line continuation) in Python (split up a long line of source code)?
- How to skip, if starts with, but match other strings
- Python dataframe optimization with two variables
- Write on image using PIL/PILLOW
- How to install matplotlib
- How do I check if a string represents a number (float or int)?
- What is the difference between a string and a byte string?
- Creating a dictionary that writes high scores to a file
- What would a "frozen dict" be?
- Remove Duplicate Dictionaries from a list based on certain numeric value
- How do I save highscore into a file with python?
- How can I use setuptools to create an exe launcher
- Python streamlit realtime speech-to-text with azure SDK
- Create per-instance property descriptor?
- saving cProfile results to readable external file
- pathlib.Path vs. os.path.join in Python
- Why does declaring a descriptor class in the __init__ function break the descriptor functionality?