I am trying to scrape bookings data from a website. Upon entering the site, I must:
Using Selenium, Python, and BeautifulSoup, when I execute Selenium commands to update the fields for the date range, and click the 'Refresh' button, by the time the code reaches the line where I extract HTML data using BeautifulSoup, the default bookings data for the default website date range is extracted, and not the bookings data for the date range I filled in.
Upon observation, it appears the site is moving by too quickly with Selenium to extract the updated data which takes some time to load. I tried to solve this using driver.implicitly_wait(5)
but this returns highly inconsistent results.
I wanted to detect an HTML element that allows for stable extraction with no errors. I realized that when I click the 'Refresh' button, this particular element ...
<div id="textArea" style="visibility: visible; display: block;">
... changes to ...
<div id="textArea" style="visibility: hidden; display: block;">
... and changes back to ...
<div id="textArea" style="visibility: visible; display: block;">
... when the bookings data on the page is updated.
Is there a Selenium command that can detect this change prior to moving onto the next lines in my Python code? For instance, after clicking the 'Refresh' button, a "wait until style=visibility is visible" command?
while True:
a = driver.find_element_by_id('textArea').get_attribute('style')
if a == r"visibility: hidden; display: block;"
break
else:
time.sleep(.05)
while True:
a = driver.find_element_by_id('textArea').get_attribute('style')
if a == r"visibility: visible; display: block;"
break
else:
time.sleep(.05)
This will wait until it goes invisible, then to visible before your program does anything