[SOLVED] Fetch all href link using selenium in python

Fetch all href link using selenium in python

I am practicing Selenium in Python and I wanted to fetch all the links on a web page using Selenium.

For example, I want all the links in the href= property of all the <a> tags on http://psychoticelites.com/

I've written a script and it is working. But, it's giving me the object address. I've tried using the id tag to get the value, but, it doesn't work.

My current script:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys


driver = webdriver.Firefox()
driver.get("http://psychoticelites.com/")

assert "Psychotic" in driver.title

continue_link = driver.find_element_by_tag_name('a')
elem = driver.find_elements_by_xpath("//*[@href]")
#x = str(continue_link)
#print(continue_link)
print(elem)

Solution

For Selenium >4.3.0, you can try the following:

from selenium.webdriver.common.by import By

elems = driver.find_elements(by=By.XPATH, "//a[@href]")
for elem in elems:
    print(elem.get_attribute("href"))

You can also refer to answers here for a more detailed explanation.

Note: Below solution works for Selenium <4.3.0.

Well, you have to simply loop through the list:

elems = driver.find_elements_by_xpath("//a[@href]")
for elem in elems:
    print(elem.get_attribute("href"))

find_elements_by_* returns a list of elements (note the spelling of 'elements'). Loop through the list, take each element and fetch the required attribute value you want from it (in this case href).