pythonpython-3.xiota

How extract URL from web page

I need a way to extract url from the list at this web page https://iota-nodes.net/ using Python. I tried BeautifulSoup but without success. My code is:

from bs4 import BeautifulSoup, SoupStrainer
import requests

url = "https://iota-nodes.net/"

page = requests.get(url)    
data = page.text
soup = BeautifulSoup(data)

for link in soup.find_all('a'):
   print(link.get('href'))
Solution

  • No need for BeautifulSoup, as the data is coming from an AJAX request. Something like this should work:

    import requests

response = requests.get('https://api.iota-nodes.net/')
data = response.json()

hostnames = [node['hostname'] for node in data]

    Note that the data comes from an API endpoint being https://api.iota-nodes.net/.