beautifulsoupnltkminingedgar

I want to parse multiple HTML documents with beautiful soup but I can't make it work


Is there a way to use beautiful soup to parse multiple HTML documents at the same time? I am modifying the code online that extracts HTML.txt files from edgar with beautiful soup so they can be downloaded as formated files: however, I found that my codes now only prints one edgar document (it's intended to print 5) and I don't know what's wrong with it.

import csv
import requests
import re
from bs4 import BeautifulSoup 

with open('General Motors Co 11-15.csv', newline='') as csvfile:
    reader = csv.reader(csvfile, delimiter=',')
    for line in reader:
        fn1 = line[0]
        fn2 = re.sub(r'[/\\]', '', line[1])
        fn3 = re.sub(r'[/\\]', '', line[2])
        fn4 = line[3]
        saveas = '-'.join([fn1, fn2, fn3, fn4])
        # Reorganize to rename the output filename.
        url = 'https://www.sec.gov/Archives/' + line[4].strip()
        bodytext=requests.get(url).text 
        parsedContent=BeautifulSoup(bodytext, 'html.parser')
        for script in parsedContent(["script", "style"]): 
            script.extract()
        text = parsedContent.get_text()
        lines = (line.strip() for line in text.splitlines())
        chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
        text = '\n'.join(chunk for chunk in chunks if chunk) 
        with open(saveas, 'wb') as f:
            f.write(requests.get('%s' % text).content)
            print(file, 'downloaded and wrote to text file')

Do you know what's wrong with my codes?


Solution

  • I would guess that you're overwriting the existing document every time you write to the file. trying changing with open(saveas, 'wb') as f: to with open(saveas, 'ab') as f:

    opening a file as wb creates a new document at with the same name as saveas, essentially clearing the existing document.