How to replay HTTP requests in Python

I would like to replay an arbitrary raw HTTP request in python. As an example, lets use an arbitrary GET request from my chromium browser to Google:

GET / HTTP/1.1
Host: google.de
Cookie: CONSENT=PENDING+071; AEC=ARSKqsKvfEvPS9Vp1bDM6YMPdHpVOoH357W-7q3cqtdDwYeHf3MqPEO1xA; SOCS=CAISHAgBEhJnd3NfMjAyMzAyMjMtMF9SQzEaAmRlIAEaBgiAn-WfBg; NID=511=N6YvXcWd_hnVVnV8w6JK4jscqE2pEt8MuTrw3yZJp-84ZxV6RJLee_yj2DEo2UJuOse0sqLjdnAD7qgPw9al7aEJqsQOCAQPIs21rLy5HQ5IAoObj7icI7ayKJttejI9Va2jDFkk0ZLvUC7P_VPJuxRJyhvLspqU1YVUcYCThrYizbo; 1P_JAR=2023-2-25-20
Sec-Ch-Ua: "Not A(Brand";v="24", "Chromium";v="110"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Linux"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.5481.78 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
X-Client-Data: CO3/ygE=
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Connection: close

Since I want my request to look exactly like the above, an obvious way would be to use the socket library:

def send(host, port, message):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((host, port))
        s.sendall(message)
        print("%s" % message.decode('us-ascii'))
        data = s.recv(1024)

    print("%s" % data)

However, that way, I obviously can't use TLS or proxies (without writing my own libraries).

So, I thought the request library could be worth a try since it already supports TLS and proxies out of the box. But I can't figure out how to load my HTTP request that I want to replay. I suppose the library was not designed for that.

I could try something like this (from the docs):

from requests import Request, Session

s = Session()

req = Request('POST', url, data=data, headers=headers)
prepped = req.prepare()

# do something with prepped.body
prepped.body = 'No, I want exactly this as the body.'

# do something with prepped.headers
del prepped.headers['Content-Type']

resp = s.send(prepped,
    stream=stream,
    verify=verify,
    proxies=proxies,
    cert=cert,
    timeout=timeout
)

print(resp.status_code)

but I still would have to first parse the headers of my request manually. If there a library that takes care of that?

Thanks for your input!

Solution

TLS can be used with sockets; you just need to wrap it in ssl library sockets.

However, there are many libraries that can do it, so you don't have to use TLS yourself.

You can use the aioreq library, which is a simple asynchronous HTTP client that can be used to send raw HTTP request messages.

Aioreq does not support proxies for raw requests, but it does allow you to do many interesting things and see how the HTTP protocol works.

This is your request (without the Cookie header, it was quite large:D).

import asyncio
import aioreq
import socket

async def main():
    hostname = "google.de"
    transport = aioreq.Transport()
    await transport.make_connection(
        ip=socket.gethostbyname(hostname),
        port=443,
        ssl=True,  # True if you want to use SSL/TLS
        server_hostname=hostname
    )

    raw_request_bytes = ("GET / HTTP/1.1\r\n"
"Host: google.de\r\n"
'Sec-Ch-Ua: "Not A(Brand";v="24", "Chromium";v="110"\r\n'
"Sec-Ch-Ua-Mobile: ?0\r\n"
'Sec-Ch-Ua-Platform: "Linux"\r\n'
"Upgrade-Insecure-Requests: 1\r\n"
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                         "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.5481.78 Safari/537.36\r\n"
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,'
                         'image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7\r\n'
"X-Client-Data: CO3/ygE=\r\n"
"Sec-Fetch-Site: none\r\n"
"Sec-Fetch-Mode: navigate\r\n"
"Sec-Fetch-User: ?1\r\n"
"Sec-Fetch-Dest: document\r\n"
"Accept-Encoding: gzip, deflate\r\n"
"Accept-Language: en-US,en;q=0.9\r\n"
"Connection: close\r\n\r\n"
).encode("ascii")

    status_line, headers_line, content = await transport.send_http_request(
        raw_request_bytes
    )
    resp = aioreq.parsers.ResponseParser.parse(status_line, headers_line, content)
    print(resp)
asyncio.run(main())

As a result, we have the variable resp, which is our HTTP Response with all of its required attributes, such as (.headers, .content, .status, ...)