python-3.xsanitizefeedparserstringio

How do I pass raw untrusted text to feedparser.parse method in Python?


So here is my code where I am trying to wrap it into StringIO class

import feedparser
import io

def read():
    import os
    name = os.path.join(os.getcwd(), 'extras', 'feeds',
                        'zycrypto.com_1596955288219')
    f = open(name, "r")
    text = f.read()
    f.close()
    return text

text = read()
parsed = feedparser.parse(io.StringIO(text))
for i in parsed.entries:
    print(i.summary, '\n')

However I keep getting this error

Traceback (most recent call last):
  File "./server/python/test.py", line 14, in <module>
    parsed = feedparser.parse(io.StringIO(text))
  File "/Users/zup/.local/share/virtualenvs/myapp_v3-kUGnE3_O/lib/python3.7/site-packages/feedparser.py", line 3922, in parse
    data, result['encoding'], error = convert_to_utf8(http_headers, data)
  File "/Users/zup/.local/share/virtualenvs/myapp_v3-kUGnE3_O/lib/python3.7/site-packages/feedparser.py", line 3574, in convert_to_utf8
    xml_encoding_match = RE_XML_PI_ENCODING.match(tempdata)
TypeError: cannot use a bytes pattern on a string-like object

Solution

  • Apparently feedparser.parse internally expects a bytes object where it is currently receiving a string, because it passes that object to a regex matching function where it uses a bytes pattern, and the object to match and the pattern need to have the same type.

    You can get a bytes object by changing open(..., 'r') to open(..., 'rb') and using BytesIO instead of StringIO.