pythonpython-3.xpython-requestsurllibsiteminder

Using Python Requests With Siteminder


I am having some trouble using Requests to retrieve some data from a webpage. It uses Siteminder and the initial form only has three fields but when I submit it, my password is changed to hex and other fields are added. Can't seem to get it to work at all. I keeps returning an error message.

Any help is appreciated and I apologize for the long post!

Edit: included the two JavaScript functions because they change the data.

Python:

from requests import session
with session() as s:

    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
    }
    payload = {
        'USER': 'username',
        'PASSWORD': 'pw',
        'TARGET': 'https://www.THISSITE.com/pg'
    }

    resp = s.post('https://www.THISSITE.com/THIS/fcc/THISSITE.fcc', headers=headers, data=payload )
    html = resp.text
    print(html)

Form:

<form
id="login"
method="post"
name="Login"
action="https://www.THISSITE.com/THIS/fcc/THISSITE.fcc">
<input
    type="hidden"
    name="TARGET"
    value="https://www.THISSITE.com/pg"
></input>
<div class="form-group">
    <input
        type="text"
        id="USER"
        name="USER"
        value=""
    ></input>
<div class="form-group">
    <input
        type="password"
        id="PASSWORD"
        name="PASSWORD"
        value=""
    ></input>
</div>
<input
    type="submit"
    name="OK"
    value="Login"
    onclick="submitAuthForm(this.form);"
></input>

submitAuthForm(form):

function submitAuthForm(form) {

    var strval = form.PASSWORD.value;

    if(!isJson(strval)){
        var info = {};
        info["Password"] = hexEncode(strval);
        form.PASSWORD.value = JSON.stringify(info);
    }
}

hexEncode(str):

function hexEncode(s){

    var chrsz   = 8;
    var hexcase = 0;

function str2binb (str) {
        var bin = Array();
        var mask = (1 << chrsz) - 1;
        for(var i = 0; i < str.length * chrsz; i += chrsz) {
            bin[i>>5] |= (str.charCodeAt(i / chrsz) & mask) << (24 - i%32);
        }
        return bin;
    }

    function Utf8Encode(string) {
        string = string.replace(/\r\n/g,"\n");
        var utftext = "";

        for (var n = 0; n < string.length; n++) {

            var c = string.charCodeAt(n);

            if (c < 128) {
                utftext += String.fromCharCode(c);
            }
            else if((c > 127) && (c < 2048)) {
                utftext += String.fromCharCode((c >> 6) | 192);
                utftext += String.fromCharCode((c & 63) | 128);
            }
            else {
                utftext += String.fromCharCode((c >> 12) | 224);
                utftext += String.fromCharCode(((c >> 6) & 63) | 128);
                utftext += String.fromCharCode((c & 63) | 128);
            }

        }

        return utftext;
    }

    function binb2hex (binarray) {
        var hex_tab = hexcase ? "0123456789ABCDEF" : "0123456789abcdef";
        var str = "";
        for(var i = 0; i < binarray.length * 4; i++) {
            str += hex_tab.charAt((binarray[i>>2] >> ((3 - i%4)*8+4)) & 0xF) +
            hex_tab.charAt((binarray[i>>2] >> ((3 - i%4)*8  )) & 0xF);
        }
        return str;
    }

    s = Utf8Encode(s);
    return binb2hex(str2binb(s));
}

Parameters when I submit via webpage:

SMENC: UTF-8
SMLOCALE: US-EN
target: https://www.THISSITE.com/pg
smauthreason: 27
smagentname: mR2h1e4BPUPZ5eTpyZckvJXpXO1mE5RpNTYtnh9C8sMfqiHlbrnBjW2SNjbwIRz+
type: 
realmoid: 
smusermsg: 
USER: username
PASSWORD: {"TokenId":"longstringoflettersandnumbersHEX???","Password":""}

Solution

  • The hexEncode function is taking a string and converting to a series of hex representations of the constituent bytes of it's UTF8 encoded representataion. The equivalent in Python would be to encode an input unicode string in UTF-8 and then re-encode the result of that in hex, e.g.

    >>> import binascii
    >>> binascii.hexlify('d'.encode('utf-8'))
    b'64'
    
    >>> binascii.hexlify('¡¢£¤¥'.encode('utf-8'))
    b'c2a1c2a2c2a3c2a4c2a5'
    

    Note: in Python 2.7 this would be —

    >>> 'd'.encode('utf-8').encode('hex')
    '64'
    
    >>> u'¡¢£¤¥'.encode('utf-8').encode('hex')
    'c2a1c2a2c2a3c2a4c2a5'
    

    If you test that with your example password, it should produce the same output as for the website, with one caveat.

    hexEncode('d')
    "64000000"
    

    Notice that the Javascript adds a number of trailing 0's, making the length of the string a multiple of 8. We need to pad the result we have to get the same output.

    >>> s = binascii.hexlify('d'.encode('utf-8'))
    >>> n = len(s)
    >>> from math import ceil
    >>> next_8_multiple = int(ceil(n/8.0) * 8)
    >>> s.ljust(next_8_multiple, b'0')
    b'6400000000'
    

    We can wrap that up in a complete function:

    from math import ceil
    import binascii
    
    def hex_encode_and_pad(s):
        hex = binascii.hexlify(s.encode('utf-8'))
        n = len(hex)
        next_8_multiple = int(ceil(n/8.0) * 8)
        zeros_to_append = next_8_multiple - n
        return hex.ljust(next_8_multiple, b'0')
    

    This now gives the same result as for the Javascript function:

    >>> hex_encode_and_pad('d')
    '64000000'
    

    The next step would be to wrap it in a string representation of the JSON. You can either do that by hand-coding the string + just interpolating the token, e.g.

    value = '{"TokenId":"%s","Password":""}' % token
    

    Or by creating the JSON string from a Python dictionary —

    import json
    data = {'TokenId': token, 'Password': ''}
    value = json.dumps(data)
    

    The complete code based on the example request shown above would be:

    import binascii
    import json
    from math import ceil
    from requests import session
    
    def hex_encode_and_pad(s):
        hex = binascii.hexlify(s.encode('utf-8'))
        n = len(hex)
        next_8_multiple = int(ceil(n/8.0) * 8)
        zeros_to_append = next_8_multiple - n
        return hex.ljust(next_8_multiple, b'0')
    
    with session() as s:
    
        password = u'your_password'
    
        token = hex_encode_and_pad(password)
        data = {'TokenId': token, 'Password': ''}
    
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'
        }
        payload = {
            'USER': 'username',
            'PASSWORD': json.dumps(data),
            'TARGET': 'https://www.THISSITE.com/pg'
        }
    
        resp = s.post('https://www.THISSITE.com/THIS/fcc/THISSITE.fcc', headers=headers, data=payload )
        html = resp.text
        print(html)