pythonpython-3.xpython-requestsproxy-server

How to use a proxy in python requests, when the password for the proxy contains a question mark?


I need to get some data using an API.

I need something like:

response = requests.get(url)

which works only locally.

Otherwise, I'm getting following error:

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')

I found out that this can be resolved by using proxies.

So I tried first with my private credentials:

response = requests.get(url, proxies={"https": "http://MY_USERNAME:MY_PASSWORD@proxyserver:port"})

That works both locally and in prod.

However, instead of private credentials, I need to use some service credentials. The problem is password contains a question mark "?".

When I run

response = requests.get(url, proxies={"https": "http://SERVICE_USERNAME:SERVICE_PASSWORD@proxyserver:port"})

I get an error:

requests.exceptions.InvalidURL: Failed to parse: http://SERVICE_USERNAME:SERVICE_PASSWORD@proxyserver:port

I also tried the approach suggested in the requests doc where I exported "http://SERVICE_USERNAME:SERVICE_PASSWORD@proxyserver:port" to a variable HTTPS_PROXY.

However, I got same error as without proxies.


Solution

  • I never tried proxy with password but try this code below and give me the result

    import requests
    from urllib import parse
    
    url = 'http://example.com'
    
    password = 'pass???word!!!'
    username = 'username'
    
    proxyserver = 'proxyserver'
    port = '8080'
    
    
    password = parse.quote(password) 
    username = parse.quote(username)
    response = requests.get(url, proxies={"https": f"http://{username}:{password}@{proxyserver}:{port}"})