pythonweb-scrapingbeautifulsoup

Use Beautiful Soup to count Title/Links

I am attempting to write a code that keeps track of the text for the links in the left handed gray box on this webpage. In this case the code should return

Valykrie, The
Acid Baby

Here is the code I am trying to use:

import requests
from bs4 import BeautifulSoup

url = 'https://www.mountainproject.com/area/109928429/aasgard-sentinel'
page = requests.get(url)
soup = BeautifulSoup(page.text, "html.parser")

for link in soup.findAll('a', class_= 'new-indicator'):
    print(link)

It is not working (otherwise I wouldn't be here!) I'm pretty new to BeautifulSoup, and coding in general. No matter how much I inspect the page source I can't seem to figure out the inputs to the findAll to get it to return what I want!

Solution

  • Search for the table with a specific I’d, than the rows:

    import requests
from bs4 import BeautifulSoup

url = 'https://www.mountainproject.com/area/109928429/aasgard-sentinel'
page = requests.get(url)
soup = BeautifulSoup(page.text, "html.parser")

table = soup.find(lambda tag: tag.name=='table' and tag.has_attr('id') and tag['id']=="left-nav-route-table") 
rows = table.findAll(lambda tag: tag.name=='a')

for member in rows:
    print(member.text)

    Output:

    Acid Baby
Valkyrie, The

    If you would also see the href link add

    print(member.get('href'))