I am learning selenium. I am trying to scrape the amazon website with selenium. Here is the link I am trying to scrape.
In the above url I am trying to extract all the elements with the class a-size-mini
and extract the link from these elements.
here is my code
links = driver.find_elements_by_class_name("a-size-mini")
for link in links:
element = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.LINK_TEXT, link.text)))
print(element.get_attribute('href'))
But this is returning None
. I am not sure what I am doing wrong. the length of the links
list is showing as 55 and when I try to print the element
variable I get the following
<selenium.webdriver.remote.webelement.WebElement (session="121606058bd493d1a70fc957699d7f6d", element="c3dd6f5b-a9bb-409c-8ee2-666cac7e7432")>
So these variables are not empty or None. But when I try to extract the link using get_attribute('href')
method it returns None
Please help me out. Thanks in advance
Please use this command.
links = driver.find_elements_by_xpath('//h2[contains(@class, "a-size-mini")]/a')
It's more efficient to parse html by xpath than class name.