I am building a web scraper which loops through a list of addresses and searches them on a property website. It then updates some dropdowns based on information we already know about the properties before scraping various information such as the expected rental yield.
It can take some time for the required elements (eg, 'bathrooms_dropdown') to load on the site once each address has been searched. I have been managing this using time.sleep(x) but this is slow and unreliable, and the implictly_wait(60) does not appear to be having any effect as I am still often getting an 'element does not exist / could not be found' error.
I am sure I need to implement WebDriverWait but cannot work out the syntax when implementing it into my code. I have seen no examples of this being used in conjunction with driver.find_elements()
Any help would be greatly appreciated!
driver.get(url)
driver.implicitly_wait(60)
# find search bar and search for address
searchBar = driver.find_element(by = By.ID, value = 'react-select-3-input')
searchBar.send_keys(address)
searchButton = driver.find_element(By.CLASS_NAME, value='sc-1mx0n6y-0').click()
# wait for elements to load
time.sleep(3) # REPLACE THIS
# find dropdown and click to open it
bathrooms_dropdown = driver.find_elements(By.CLASS_NAME, value = 'css-19bqh2r')[-2]
bathrooms_dropdown.click()
You need to take care of a couple of things here as follows:
As the searchBar element is a clickable element ideally you need you need to induce WebDriverWait for the element_to_be_clickable() as follows:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "react-select-3-input"))).send_keys(address)
Again as the searchButton element is a clickable element you need you need to induce WebDriverWait for the element_to_be_clickable() as follows (in the worst case scenario assuming the searchButton gets enabled when search text is populated):
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CLASS_NAME, "sc-1mx0n6y-0"))).click()
Ideal dropdowns are html-select tags and ideally you should be using the Select class inducing WebDriverWait as follows:
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "cssSelector_select_element")))).select_by_visible_text("visible_text")
Finally, the ID and CLASS_NAME values which you have used e.g. react-select-3-input, sc-1mx0n6y-0, css-19bqh2r, etc looks dynamic and may change when you would access the application afresh or in short intervals. So you may opt to lookout for some other static attributes.
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC