pythonpython-3.xgoogle-chromeweb-scrapinginspect-element

Web Scraping "Inspect" Element section in RatemyProfessor Website

I'm relatively new to python and wanted to see if there is any means to scrap the inspect Element section of the RatemyProfessor site. My goal is to obtain all the professor ID's which are only located in that area.

When attempting to obtain the code I tried..

import requests

r = requests.get('http://www.ratemyprofessors.com/search.jsp?queryBy=schoolId&schoolName=California+State+University%2C+Northridge&schoolID=163&queryoption=TEACHER')

print (r.text)

But unfortunately only received the source page information, which doesn't provide the id information. The id's are located in the Inspect Element section, and I was wondering if there is a special link I'm just not seeing that would help me extract this data

This is for a college project, if anyone was curious, any suggestions will help!

Thanks again!

UPDATE Thank you for all the feedback I really appreciate it, but i'm still not understanding the logic of how I would be able to obtain the information of the elements with the link of the source code

Here I placed arrows indicating what i'm seeing, the link in my "requests.get" provides the code on the left, and my goal is to find a url, or something to be able to extract the information which is on the right.

I really want to understand what is going on, and the proper way to approach this, if someone can explain this to me the process of how this can be achieved I would greatly appreciate it.

Once again thank you everyone for contributing I really appreicate it!

Solution

  • I did not test, but you can use the lib beautifulSoup to parse the hml code, and after that find all div with class 'result-list' and make a find_all with all 'li' html code. Now you can get the id of that li, split the result and get the last position. Something like that:

    import requests
from bs4 import BeautifulSoup

r = requests.get('http://www.ratemyprofessors.com/search.jsp?queryBy=schoolId&schoolName=California+State+University%2C+Northridge&schoolID=163&queryoption=TEACHER')
page = BeautifulSoup(r.content, 'html.parser')
for divtag in soup.find_all('div', {'class': 'result-list'}):
    for litag in ultag.find_all('li'):
        print litag.text

    I dit not test my code, but the logic is that.