I am trying to use Python to web scrape a website that loads it's HTML dynamically by using embedded javascript files that render the data as a Response into the HTML. Therefore, if I use BeautifulSoup alone, I will not be able to retrieve that data that I need as my program will scrape it before the Javascript loads the data. Due to this, I am integrating the selenium library into my code, to make my program wait until a certain element is found before it scrapes the website.
I had originally done this:
element = WebDriverWait(driver,100).until(EC.presence_of_element_located((By.ID, "tabla_evolucion")))
But I want to specify a class instead by doing something like:
element = WebDriverWait(driver,100).until(EC.presence_of_element_located((By.class, "ng-binding ng-scope")))
Here is the rest of my code:
driver_path = 'C:/webDrivers/chromedriver.exe'
driver = webdriver.Chrome(executable_path=driver_path)
driver.header_overrides = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'}
url = "myurlthatIamscraping.com"
response = driver.get(url)
html = driver.page_source
characters = len(html)
element = WebDriverWait(driver,100).until(EC.presence_of_element_located((By.class, "ng-binding ng-scope")))
print(html)
print(characters)
time.sleep(10)
driver.quit()
It is not working for me and I can not find the right syntax anywhere.
The relevant HTML would have helped us to construct a more canonical answer. However to start with your first line of code:
element = WebDriverWait(driver,100).until(EC.presence_of_element_located(
(By.ID, "tabla_evolucion")))
is pretty much legitimate where as the second line of code:
element = WebDriverWait(driver,100).until(EC.presence_of_element_located(
(By.class, "ng-binding ng-scope")))
Will raise an error as:
Message: invalid selector: Compound class names not permitted
as you can't pass multiple classes through By.class.
You can find a detailed discussion in Invalid selector: Compound class names not permitted using find_element_by_class_name with Webdriver and Python
You need to take care of a couple of things as follows:
presence_of_element_located() merely confirms the presence of the element within the DOM Tree. Presumably moving ahead either you need to get the attributes e.g. value, innerText, etc or you would interact with the element. So instead of presence_of_element_located() you need to use either visibility_of_element_located() or element_to_be_clickable()You can find a detailed discussion in WebDriverWait not working as expected
For an optimum result you can club up the ID and CLASS attributes and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located(
(By.CSS_SELECTOR, ".ng-binding.ng-scope#tabla_evolucion")))
XPATH: element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located(
(By.XPATH, "//*[@class='ng-binding ng-scope' and @id='tabla_evolucion']")))