pythonpython-3.xweb-scrapingpython-requests

Unable to get a response after sending an HTTP request to a website


I'm trying to send an HTTP request to a website using the requests module, expecting to get a 200 status code. However, when I run the script, I always receive an error, which I will paste below. I've added the headers, but I still haven't received any response.

import requests

link = 'https://apps.hcr.ny.gov/BuildingSearch/'

headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'accept-language': 'en-US,en;q=0.9',
    'host': 'apps.hcr.ny.gov',
    'origin': 'https://apps.hcr.ny.gov',
    'referer': 'https://apps.hcr.ny.gov/BuildingSearch/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
}

with requests.Session() as s:
    s.headers.update(headers)
    res = s.get(link,headers=headers)
    print(res.status_code)

I receive the following error:

  File "C:\Users\CL\AppData\Local\Programs\Python\Python312\Lib\site-packages\requests\adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Solution

  • The website you're accessing is not RFC 7230 compliant. The server appears to be Microsoft-IIS/10.0

    HTTP headers should be treated as case-insensitive.

    Here's the proof:

    import requests
    import json
    
    URL = 'https://apps.hcr.ny.gov/BuildingSearch/'
    UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15"
    
    for key in "user-agent", "User-Agent":
        headers = {
            key: UA
        }
        print("\nTrying with:", json.dumps(headers, indent=2))
        try:
            with requests.get(URL, headers=headers) as response:
                response.raise_for_status()
                print("Success")
        except Exception:
            print("Failed")
    

    Output:

    Trying with: {
      "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15"
    }
    Failed
    
    Trying with: {
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_7_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15"
    }
    Success