pythonweb-scrapinghyperlinkwikipedia

How to scrape links from Wikipedia with Python


I am trying to scrape all the Links to battles from the "List of Naval Battles" on Wikipedia using python. The trouble is that I cannot figure out how to export all of the links containing the words "/wiki/Battle" to my CSV file. I am used to C++, so python is kind of foreign to me. Any ideas? Here is what I have so far...

from bs4 import BeautifulSoup
import urllib2

rootUrl = "https://en.wikipedia.org/wiki/List_of_naval_battles"


def get_soup(url,header):
    return
BeautifulSoup(
    urllib2.urlopen(urllib2.Request(url,headers=header)),'html.parser')

# soup settings    
url = rootUrl + item
header={'User-Agent':"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36"}

soup = get_soup(url,header)

battle = soup.findAll("/wiki/Battle")

Solution

  • Try this:

    from bs4 import BeautifulSoup as bs
    import requests
    
    res = requests.get("https://en.wikipedia.org/wiki/List_of_naval_battles")
    soup = bs(res.text, "html.parser")
    naval_battles = {}
    for link in soup.find_all("a"):
        url = link.get("href", "")
        if "/wiki/Battle" in url:
            naval_battles[link.text.strip()] = url
    
    print(naval_battles)