pythonselenium-webdriverloadingwait

Python Selenium. Wait for page fully loaded


I am trying to implement website automation. Before I start to interact with website, I have to wait until page is fully loaded. The website work with great amount of data and fetching it from database. So the loading takes few seconds (depends on internet speed).

I tried to use time.sleep(10), but is not stable. How I said before, the loading depends of internet speed. So, one time it works, another one not.

Also i tried:

wait.until(
    lambda driver: driver.execute_script("return document.readyState") == "complete"
)

It didn't work. It looks for DOM to be appeared, but DOM is changing right as data is fetched from database.

Explicit wait is also not a solution. On website i have multiple pages. Navigating on any of them requires time for loading. There are no same elements on all pages. So I can't rely on this solution.

I need universal waiter for page to be fully loaded with fetched data. How to make it? I didn't found answer on internet.


Solution

  • A practical universal trick: wait until the DOM stops changing for a few seconds.

    import time
    from selenium.webdriver.support.ui import WebDriverWait
    
    def wait_for_dom_stability(driver, timeout=30, stable_time=2):
        end_time = time.time() + timeout
        last_source = None
        stable_start = None
    
        while time.time() < end_time:
            current_source = driver.page_source
            if current_source == last_source:
                if stable_start is None:
                    stable_start = time.time()
                elif time.time() - stable_start >= stable_time:
                    return True
            else:
                stable_start = None
            last_source = current_source
            time.sleep(0.5)
        raise TimeoutError("Page did not stabilize")
    

    This doesn’t depend on framework (jQuery/fetch/etc.) and works even for SPAs, but may be a bit slower.