pythonseleniumweb-crawlerwebdriverwaitexpected-condition

Selenium find_elements only works if time passes since get()


Running these 2 lines together in Colab returns []:

wd.get("https://wetransfer.com/")
wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')

However, running one, followed by the other returns the expected result:

[<selenium.webdriver.remote.webelement.WebElement (session="3cdfb3afbb591862e909cd406b6ac523", element="19fd31e8-710a-4b6e-8284-9a7409f12718")>,
<selenium.webdriver.remote.webelement.WebElement (session="3cdfb3afbb591862e909cd406b6ac523", element="837097d1-5735-4b24-9cb2-9d3ded3a0311")>]

Get is supposed to be blocking so not sure what is going on here.


Solution

  • This is how basically Selenium works.
    It can access web elements only after they are completely loaded.
    This is why we use implicitly and explicitly waits here.
    The explicitly waits are much more recommended.
    So instead of

    wd.get("https://wetransfer.com/")
    wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')
    

    You should use something like this:

    wd.get("https://wetransfer.com/")
    wait = WebDriverWait(wd, 20)
    wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "input[type=file]")))
    wd.find_elements(By.CSS_SELECTOR, 'input[type=file]')
    

    To use it you will have to import these imports:

    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC