[SOLVED] requests.get consistent "IncompleteRead" for url

requests.get consistent "IncompleteRead" for url

When using requests.get as follows, I'm consistently getting an IncompleteRead error:

response = requests.get("https://files.rcsb.org/header/6TAV.pdb")

Tried, but did not work:

changing from https to http based on (How to handle IncompleteRead: in python) and still get IncompleteRead
setting stream=True, still get IncompleteRead

What works:

accessing the URL directly
subprocess + wget

Is there a recommended way to successfully download this URL/handle the incomplete read within python?

Solution

Try adding a suitable header in your request i.e.

headers = {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.8',
'Cache-Control': 'max-age=0',
'Connection': 'keep-alive',
'Sec-Fetch-Dest': 'document',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-Site': 'cross-site',
'Sec-Fetch-User': '?1',
'Sec-GPC': '1',
'Upgrade-Insecure-Requests': '1',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',
'sec-ch-ua': '"Chromium";v="130", "Brave";v="130", "Not?A_Brand";v="99"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"'}

response = requests.get('https://files.rcsb.org/header/6TAV.pdb', headers=headers)