pythonhtmlseleniumselenium-webdriver

Selenium Wait Until HTML Element Changes


I am trying to scrape bookings data from a website. Upon entering the site, I must:

  1. Change the default date range (at first, the site displays bookings data for the default date range) to a date range I want to search for
  2. Click the 'Refresh' button
  3. Wait a bit for the website to update the displayed results on the page.

Using Selenium, Python, and BeautifulSoup, when I execute Selenium commands to update the fields for the date range, and click the 'Refresh' button, by the time the code reaches the line where I extract HTML data using BeautifulSoup, the default bookings data for the default website date range is extracted, and not the bookings data for the date range I filled in.

Upon observation, it appears the site is moving by too quickly with Selenium to extract the updated data which takes some time to load. I tried to solve this using driver.implicitly_wait(5) but this returns highly inconsistent results.

I wanted to detect an HTML element that allows for stable extraction with no errors. I realized that when I click the 'Refresh' button, this particular element ...

<div id="textArea" style="visibility: visible; display: block;">

... changes to ...

<div id="textArea" style="visibility: hidden; display: block;">

... and changes back to ...

<div id="textArea" style="visibility: visible; display: block;">

... when the bookings data on the page is updated.

Is there a Selenium command that can detect this change prior to moving onto the next lines in my Python code? For instance, after clicking the 'Refresh' button, a "wait until style=visibility is visible" command?


Solution

  • while True:
        a = driver.find_element_by_id('textArea').get_attribute('style')
        if a == r"visibility: hidden; display: block;"
            break
        else:
            time.sleep(.05)
    while True:
        a = driver.find_element_by_id('textArea').get_attribute('style')
        if a == r"visibility: visible; display: block;"
            break
        else:
            time.sleep(.05)
    

    This will wait until it goes invisible, then to visible before your program does anything