I need to get some data using an API.
I need something like:
response = requests.get(url)
which works only locally.
Otherwise, I'm getting following error:
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')
I found out that this can be resolved by using proxies.
So I tried first with my private credentials:
response = requests.get(url, proxies={"https": "http://MY_USERNAME:MY_PASSWORD@proxyserver:port"})
That works both locally and in prod.
However, instead of private credentials, I need to use some service credentials. The problem is password contains a question mark "?".
When I run
response = requests.get(url, proxies={"https": "http://SERVICE_USERNAME:SERVICE_PASSWORD@proxyserver:port"})
I get an error:
requests.exceptions.InvalidURL: Failed to parse: http://SERVICE_USERNAME:SERVICE_PASSWORD@proxyserver:port
I also tried the approach suggested in the requests doc where I exported "http://SERVICE_USERNAME:SERVICE_PASSWORD@proxyserver:port"
to a variable HTTPS_PROXY
.
However, I got same error as without proxies.
I never tried proxy with password but try this code below and give me the result
import requests
from urllib import parse
url = 'http://example.com'
password = 'pass???word!!!'
username = 'username'
proxyserver = 'proxyserver'
port = '8080'
password = parse.quote(password)
username = parse.quote(username)
response = requests.get(url, proxies={"https": f"http://{username}:{password}@{proxyserver}:{port}"})