pythonseleniumscreen-scrapingstaleelementreferenceexception

How to handle StaleElementReferenceException (Selenium Python)?


I am writing a scraper for https://etherscan.io/ using selenium in python. I am getting StaleElementReferenceException in my code when it clicks on a button and not finding any good source to solve this error.

This is the code

url = "https://etherscan.io/login?"
driver.implicitly_wait(30)
driver.get(url)
username = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtUserName"]')
username.send_keys('user')
password = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtPassword"]')
password.send_keys('pass')
login = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_btnLogin"]')
login.click()
more = driver.find_element_by_xpath('//*[@id="moreMegaMenu"]')
more.click()
word_cloud = driver.find_element_by_xpath('//*[@id="LI41"]/a').click()
soup = BeautifulSoup(driver.page_source, 'html.parser')
wait = WebDriverWait(driver, 10)
dropdown = driver.find_elements_by_css_selector('.dropdown-toggle')
for ele in dropdown:
    ele.click()
    time.sleep(1)
    account = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.d-block')))
    driver.execute_script("arguments[0].click();", account)
    time.sleep(1)
    driver.execute_script("window.history.go(-1)")

Error Message:

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=100.0.4896.75)
StaleElementReferenceException            Traceback (most recent call last)
<ipython-input-108-2804d338c51f> in <module>
      1 dropdown = driver.find_elements_by_css_selector('.dropdown-toggle')
      2 for ele in dropdown:
----> 3     ele.click() # this is the line where it is showing error
      4     time.sleep(1)
      5     account = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'.d-block')))

Solution

  • @cruisepandey solution is ok, but I think mine is a bit simpler. Also, in my case, his solution redirects to a wrong page (to homepage).

    url = "https://etherscan.io/login?"
    driver.implicitly_wait(30)
    driver.get(url)
    username = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtUserName"]')
    username.send_keys('user')
    password = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_txtPassword"]')
    password.send_keys('pass')
    login = driver.find_element_by_xpath('//*[@id="ContentPlaceHolder1_btnLogin"]')
    login.click()
    more = driver.find_element_by_xpath('//*[@id="moreMegaMenu"]')
    more.click()
    word_cloud = driver.find_element_by_xpath('//*[@id="LI41"]/a').click()
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    wait = WebDriverWait(driver, 10)
    
    num_elements = len(driver.find_elements_by_css_selector('.dropdown-toggle'))
    
    for i in range(num_elements):
        ele = driver.find_elements_by_css_selector('.dropdown-toggle')[i]
        ele.click()
        time.sleep(1)
        account = wait.until(EC.element_to_be_clickable((By.XPATH, f'//*[@id="content"]/div[3]/div/div/div[3]/div[{i+1}]/div/div/a[1]')))
        driver.execute_script("arguments[0].click();", account)
        time.sleep(1)
        driver.execute_script("window.history.go(-1)")
        time.sleep(1)