I'm trying to send HTTPS requests as quickly as possible. I know this would have to be concurrent requests due to my goal being 150 to 500+ requests a second. I've searched everywhere, but get no Python 3.11+ answer or one that doesn't give me errors. I'm trying to avoid AIOHTTP as the rigmarole of setting it up was a pain, which didn't even work.
The input should be an array or URLs and the output an array of the html string.
This works, getting around 250+ requests a second.
This solution does work on Windows 10. You may have to pip install
for concurrent and requests.
import time
import requests
import concurrent.futures
start = int(time.time()) # get time before the requests are sent
urls = [] # input URLs/IPs array
responses = [] # output content of each request as string in an array
# create an list of 5000 sites to test with
for y in range(5000):urls.append("https://example.com")
def send(url):responses.append(requests.get(url).content)
with concurrent.futures.ThreadPoolExecutor(max_workers=10000) as executor:
futures = []
for url in urls:futures.append(executor.submit(send, url))
end = int(time.time()) # get time after stuff finishes
print(str(round(len(urls)/(end - start),0))+"/sec") # get average requests per second
Output:
286.0/sec
Note: If your code requires something extremely time dependent, replace the middle part with this:
with concurrent.futures.ThreadPoolExecutor(max_workers=10000) as executor:
futures = []
for url in urls:
futures.append(executor.submit(send, url))
for future in concurrent.futures.as_completed(futures):
responses.append(future.result())
This is a modified version of what this site showed in an example.
The secret sauce is the max_workers=10000
. Otherwise, it would average about 80/sec. Although, when setting it to beyond 1000, there wasn't any boost in speed.