parsingedgar

Saving SEC 10-K annual report text to files (trouble with decoding)

I am trying to bulk-download the text visible to the "end-user" from 10-K SEC Edgar reports (don't care about tables) and save it in a text file. I have found the code below on Youtube, however I am facing 2 challenges:

  1. I am not sure if I am capturing all text, and when I print the URL from below, I receive very weird output (special characters e.g., at the very end of the print-out)

  2. I can't seem to save the text in txt files, not sure if this is due to encoding (I am entirely new to programming).

import re
import requests
import unicodedata
from bs4 import BeautifulSoup

def restore_windows_1252_characters(restore_string):
    def to_windows_1252(match):
        try:
            return bytes([ord(match.group(0))]).decode('windows-1252')
        except UnicodeDecodeError:
            # No character at the corresponding code point: remove it.
            return ''

    return re.sub(r'[\u0080-\u0099]', to_windows_1252, restore_string)

# define the url to specific html_text file
new_html_text = r"https://www.sec.gov/Archives/edgar/data/796343/0000796343-14-000004.txt"

# grab the response
response = requests.get(new_html_text)
page_soup = BeautifulSoup(response.content,'html5lib')

page_text = page_soup.html.body.get_text(' ',strip = True)

# normalize the text, remove characters. Additionally, restore missing window characters.
page_text_norm = restore_windows_1252_characters(unicodedata.normalize('NFKD', page_text)) 

# print: this works however gives me weird special characters in the print (e.g., at the very end)
print(page_text_norm)

# save to file: this only gives me an empty text file
with open('testfile.txt','w') as file:
    file.write(page_text_norm)
Solution

  • Try this. If you take the data you expect as an example, it will be easier for people to understand your needs.

    from simplified_scrapy import SimplifiedDoc,req,utils
url = 'https://www.sec.gov/Archives/edgar/data/796343/0000796343-14-000004.txt'
html = req.get(url)
doc = SimplifiedDoc(html)
# text = doc.body.text
text = doc.body.unescape() # Converting HTML entities
utils.saveFile("testfile.txt",text)