pythonseleniumhrefgetattributefindelement

Why does my .get_attribute('href') function return none on some links?

I am receiving the href links on some links and others returned None value.

I have the following snippet to retrieve the first 16 items on a page:

def loop_artikelen ():
    artikelen = driver.find_elements(By.XPATH, "//*[@id='content']/div[2]/ul/li")
    artikelen_lijst = []
    for artikel in artikelen[0:15]:
        titel = artikel.find_element(By.CLASS_NAME, 'hz-Listing-title').text
        prijs = artikel.find_element(By.CLASS_NAME, 'hz-Listing-price').text
        link = artikel.find_element(By.CLASS_NAME, 'hz-Listing-coverLink').get_attribute('href')
        #if link == "None":
        #   link = artikel.find_element(By.XPATH(".//a").get_attribute('href'))
        artikel = titel, prijs, link
        artikelen_lijst.append(artikel)

The output looks like this when i print it out

('Fiets gestolen dus voor een mooi prijsje is ie van jou', '€ 400,00', None)
('Amslod middenmoter fiets', '€ 1.500,00', None)
('Batavus damesfiets', '€ 90,00', 'https://www.marktplaats.nl/v/fietsen-en-brommers/fietsen-dames-damesfietsen/m1933195519-batavus-damesfiets')
('Time edge', '€ 700,00', 'https://www.marktplaats.nl/v/fietsen-en-brommers/fietsen-racefietsen/m1933185638-time-edge')

I tried adding a time.sleep(2) between link and artikel, but it didn't work. You can also i tried something else after "#", that didn't work either.

Who can help me?

Thanks in advance

Link to site : https://www.marktplaats.nl/q/fiets/#offeredSince:Vandaag|sortBy:SORT_INDEX|sortOrder:DECREASING|

Solution

  • This seems like a diagnostic issue, and since it seems like you're pretty good with selenium already, I'll list some more general advice for helping with these sorts of problems rather than trying to solve this specific one. Not being able to find a web element is a very common problem. Here are some things that could be wrong:

    1. The elements are not being found because your query is wrong. A lot of websites have screwy HTML because they are coded that way for whatever reason. Sometimes something that looks like a list cannot be found with a single XPath query. Also I HIGHLY recommend using CSS paths instead of XPath, almost anything that can be gotten with an XPath can be found with a CSS path and it generally yields better results.

    2. The elements are not being found because they haven't been loaded. This is because either the webpage needs to be scrolled down or just because the website hasn't finished loading. You can try increasing the sleep timer to something like 1 minute to see if that's the problem, and/or manually scroll the page down during those 60 seconds.

    I would try (2) first to see if that fixes your problem, since it is so easy to do, and only takes a minute.

    EDIT: Since you mention: "href links on some links and others returned None value," if selenium can't find the element, it will throw an exception. If it can't find the attribute, it will return None, so the problem might be that it can find the element, but can't find the href attribute. (in other words, it is finding the wrong elements) Your problem is almost certainly that some of the elements are not links at all. I would recommend printing out all the elements you get to confirm that they are all the ones you think they are. Also, use CSSpath instead of Xpath because that will probably solve your problem.