Is there a way to use beautiful soup to parse multiple HTML documents at the same time? I am modifying the code online that extracts HTML.txt files from edgar with beautiful soup so they can be downloaded as formated files: however, I found that my codes now only prints one edgar document (it's intended to print 5) and I don't know what's wrong with it.
import csv
import requests
import re
from bs4 import BeautifulSoup
with open('General Motors Co 11-15.csv', newline='') as csvfile:
reader = csv.reader(csvfile, delimiter=',')
for line in reader:
fn1 = line[0]
fn2 = re.sub(r'[/\\]', '', line[1])
fn3 = re.sub(r'[/\\]', '', line[2])
fn4 = line[3]
saveas = '-'.join([fn1, fn2, fn3, fn4])
# Reorganize to rename the output filename.
url = 'https://www.sec.gov/Archives/' + line[4].strip()
bodytext=requests.get(url).text
parsedContent=BeautifulSoup(bodytext, 'html.parser')
for script in parsedContent(["script", "style"]):
script.extract()
text = parsedContent.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = '\n'.join(chunk for chunk in chunks if chunk)
with open(saveas, 'wb') as f:
f.write(requests.get('%s' % text).content)
print(file, 'downloaded and wrote to text file')
Do you know what's wrong with my codes?
I would guess that you're overwriting the existing document every time you write to the file. trying changing with open(saveas, 'wb') as f:
to with open(saveas, 'ab') as f:
opening a file as wb
creates a new document at with the same name as saveas
, essentially clearing the existing document.