pythonurlpython-requestspercent-encoding

How to prevent python requests from percent encoding my URLs?


I'm trying to GET an URL of the following format using requests.get() in python:

http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel

#!/usr/local/bin/python

import requests

print(requests.__versiom__)
url = 'http://api.example.com/export/'
payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'}
r = requests.get(url, params=payload)
print(r.url)

However, the URL gets percent encoded and I don't get the expected response.

2.2.1
http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json

This works if I pass the URL directly:

url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel
r = requests.get(url)

Is there some way to pass the the parameters in their original form - without percent encoding?

Thanks!


Solution

  • It is not good solution but you can use directly string:

    r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')
    

    BTW:

    Code which convert payload to this string

    payload = {
        'format': 'json', 
        'key': 'site:dummy+type:example+group:wheel'
    }
    
    payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
    # 'format=json&key=site:dummy+type:example+group:wheel'
    
    r = requests.get(url, params=payload_str)
    

    EDIT (2020):

    You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .

    As I know requests also use urllib.parse.urlencode(...) for this but without safe=.

    import requests
    import urllib.parse
    
    payload = {
        'format': 'json', 
        'key': 'site:dummy+type:example+group:wheel'
    }
    
    payload_str = urllib.parse.urlencode(payload, safe=':+')
    # 'format=json&key=site:dummy+type:example+group:wheel'
    
    url = 'https://httpbin.org/get'
    
    r = requests.get(url, params=payload_str)
    
    print(r.text)
    

    I used page https://httpbin.org/get to test it.