pythonpython-3.xheaderpython-requestspyspider

Python ValueError: Invalid header name b':authority


I see the ':' is error, but I can't find a way to solve it.

ValueError: Invalid header name b':authority'

It's the error:

File "tmall.py", line 23, in get_url
response = sessions.get(url=url,headers =headers)

File "E:\python\lib\site-packages\requests\sessions.py", line 501, in get
return self.request('GET', url, **kwargs)

File "E:\python\lib\site-packages\requests\sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)

File "E:\python\lib\site-packages\requests\sessions.py", line 609, in send
r = adapter.send(request, **kwargs)

File "E:\python\lib\site-packages\requests\adapters.py", line 423, in send
timeout=timeout

File "E:\python\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)

File "E:\python\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 356, in _make_request
conn.request(method, url, **httplib_request_kw)

File "E:\python\lib\http\client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)

File "E:\python\lib\http\client.py", line 1280, in _send_request
self.putheader(hdr, value)

File "E:\python\lib\http\client.py", line 1207, in putheader
raise ValueError('Invalid header name %r' % (header,))

It's the code:

import requests
headers = {
    ':authority':'list.tmall.com',
    ':method':'GET',
    ':path':path}
sessions = requests.session();
response = sessions.get(url=url,headers =headers)

Solution

  • The author of the question understood that the beginning of his header name (the :) is the problem. It seems he insists on using the : as part of the header name. This can be accomplished by overriding what httplib accepts as legal header name using the following code:

    import httplib
        
    httplib._is_legal_header_name = re.compile(r':|\A[^:\s][^:\r\n]*\Z').match
    

    Note: Generally this may not be a good idea. The assumption is the author of the question has very specific needs and knows why this is the only/best solution.