pythonselenium

get_attribute('href') returning None


I am learning selenium. I am trying to scrape the amazon website with selenium. Here is the link I am trying to scrape.

In the above url I am trying to extract all the elements with the class a-size-mini and extract the link from these elements.

here is my code

links = driver.find_elements_by_class_name("a-size-mini")

for link in links:
    element = WebDriverWait(driver, 5).until(
                    EC.presence_of_element_located((By.LINK_TEXT, link.text)))

    print(element.get_attribute('href'))

But this is returning None. I am not sure what I am doing wrong. the length of the links list is showing as 55 and when I try to print the element variable I get the following

<selenium.webdriver.remote.webelement.WebElement (session="121606058bd493d1a70fc957699d7f6d", element="c3dd6f5b-a9bb-409c-8ee2-666cac7e7432")>

So these variables are not empty or None. But when I try to extract the link using get_attribute('href') method it returns None

Please help me out. Thanks in advance


Solution

  • Please use this command.

    links = driver.find_elements_by_xpath('//h2[contains(@class, "a-size-mini")]/a')
    

    It's more efficient to parse html by xpath than class name.