pythonurllibbearer-token

Python urllib.request.urlopen with Bearer authentication in redirected request

The following command successfully downloads an artifact file from a GitHub workflow run:

curl -L -H "Authorization: Bearer ghp_XXXX" -o arti.zip \
     https://api.github.com/repos/OWNER/REPO/actions/artifacts/ID/zip

The following Python code fails, with the same URL and authentication token:

import os, urllib.request
req = urllib.request.Request('https://api.github.com/repos/OWNER/REPO/actions/artifacts/ID/zip')
req.add_header('Authorization', 'Bearer ghp_XXXX')

with urllib.request.urlopen(req) as input:
    with open('arti.zip', 'wb') as output:
        output.write(input.read())

The Python exception is raised in urlopen():

urllib.error.HTTPError: HTTP Error 403: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

What is wrong in the Python code?

The request is redirected to another host and we can suspect that the authorization is not redirected as well. However, the Python doc for Request.add_header() says that "headers added using this method are also added to redirected requests". So it should work.

The same code with any public URL and no authentication works. I have seen many over-complicated examples online and on SO, each one explaining that some other Python library should be used, etc. Isn't there a simple way to use urllib with a Bearer Authorization?

Solution

  • You're correct in identifying that the issue likely stems from the redirect behavior and authorization header.

    Here's what's happening:

    1. The GitHub URL you're hitting (https://api.github.com/repos/OWNER/REPO/actions/artifacts/ID/zip) responds with a 302 redirect to an AWS S3 signed URL.

    2. The Authorization: Bearer ghp_XXXX header must not be sent to that S3 URL, because it’s invalid there.

    It might trigger 403 errors like you're seeing.

    Why curl works but urllib fails:

    curl follows redirects, but drops the Authorization header on cross-domain redirects by default — which is correct behavior for security.

    Python’s urllib.request.urlopen does not drop the header when following redirects to a different domain — even though add_header() is documented to re-apply headers on redirects, it does not filter them based on domain changes.

    Use requests library

    
import requests

url = 'https://api.github.com/repos/OWNER/REPO/actions/artifacts/ID/zip'

headers = {'Authorization': 'Bearer ghp_XXXX'}



response = requests.get(url, headers=headers, allow_redirects=True)

with open('arti.zip', 'wb') as f:

    f.write(response.content)

    The requests library does the right thing automatically: it drops Authorization headers when redirecting across domains.