pythonseleniumxpathwebdriverwaitpubchem

Getting the content of a table on the website with Selenium and Python


When I go to the web address in the code I don't get the contents from "Synonyms" section. It does the selection, but takes it as a list and does not output the text content.

synonyms= []
driver= webdriver.Chrome()
url = "https://pubchem.ncbi.nlm.nih.gov/compound/71308229"
driver.get(url)
synonym = driver.find_elements_by_class_name("overflow-x-auto")
synonyms.append(synonym)
driver.close()

Solution

    1. You are missing a wait / delay.
    2. You have to extract the text from the web element(s)
    3. Looks like you are using a wrong locator

    I guess this will give you what you are looking for:

    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    import time
    
    synonyms= []
    driver= webdriver.Chrome()
    url = "https://pubchem.ncbi.nlm.nih.gov/compound/71308229"
    driver.get(url)
    wait = WebDriverWait(driver, 20)
    wait.until(EC.visibility_of_element_located((By.XPATH, "//div[@class='overflow-x-auto']//p")))
    time.sleep(0.1)
    elements = driver.find_elements_by_xpath("//div[@class='overflow-x-auto']//p")
    for el in elements:
        synonyms.append(el.text)
    driver.close()