pythonpython-3.xhttphttpsconcurrency

How do I get Python to send as many concurrent HTTP requests as possible?


I'm trying to send HTTPS requests as quickly as possible. I know this would have to be concurrent requests due to my goal being 150 to 500+ requests a second. I've searched everywhere, but get no Python 3.11+ answer or one that doesn't give me errors. I'm trying to avoid AIOHTTP as the rigmarole of setting it up was a pain, which didn't even work.

The input should be an array or URLs and the output an array of the html string.


Solution

  • This works, getting around 250+ requests a second. This solution does work on Windows 10. You may have to pip install for concurrent and requests.

    import time
    import requests
    import concurrent.futures
    
    start = int(time.time()) # get time before the requests are sent
    
    urls = [] # input URLs/IPs array
    responses = [] # output content of each request as string in an array
    
    # create an list of 5000 sites to test with
    for y in range(5000):urls.append("https://example.com")
    
    def send(url):responses.append(requests.get(url).content)
    
    with concurrent.futures.ThreadPoolExecutor(max_workers=10000) as executor:
        futures = []
        for url in urls:futures.append(executor.submit(send, url))
            
    end = int(time.time()) # get time after stuff finishes
    print(str(round(len(urls)/(end - start),0))+"/sec") # get average requests per second
    

    Output: 286.0/sec

    Note: If your code requires something extremely time dependent, replace the middle part with this:

    with concurrent.futures.ThreadPoolExecutor(max_workers=10000) as executor:
        futures = []
        for url in urls:
            futures.append(executor.submit(send, url))
        for future in concurrent.futures.as_completed(futures):
            responses.append(future.result())
    

    This is a modified version of what this site showed in an example.

    The secret sauce is the max_workers=10000. Otherwise, it would average about 80/sec. Although, when setting it to beyond 1000, there wasn't any boost in speed.