pythonpython-requestsgrequests

Multithreading requests without a URL list


I'm currently trying to make my requests faster by multithreading them but I'm not sure how to do it the way I want. I know about grequests but they seem to require a URL list. I have code with a starting number contained in URL and would like all threads to stop after getting a status_code of 200

I have tried to accomplish this by grequests but couldn't make it work. Also tried threading but don't know how to stop all threads after working URL was found

import requests
import webbrowser

def url_request(number):
    url = "http://website.com/download/" + str(number) + ".zip"
    r = requests.head(url)
    if r.status_code == 404:
        print(url + " - 404 Not Found!")
        number += 1
        url_request(number)
    elif r.status_code == 200:
        webbrowser.open(url)
        print(url + " - 200 Found!")

if __name__ == "__main__":
    url_request(int(input("Starting number: ")))

What I want the code to do is execute multiple request.head at once with a number after "Starting number" and will stop after one of the threads finds url with status_code 200.


Solution

  • Ok, figured it out. Thanks for your advice.

    Here's the code:

    from gevent import monkey
    monkey.patch_all()
    import grequests
    import webbrowser
    
    
    def url_request_threaded(startnumber, stopnumber):
        urls = []
        for i in range(startnumber, stopnumber):
            urls.append("http://website.com/download/" + str(i) + ".zip")
        gr = (grequests.head(url, stream=False) for url in urls)
        gresponses = grequests.imap(gr, size=10)
        try:
            for response in gresponses:
                if response.status_code == 404:
                    print(response.url + " - 404 Not Found!")
                elif response.status_code == 200:
                    webbrowser.open(response.url)
                    print(response.url + " - 200 Found!")
                    raise SystemExit
        except SystemExit:
            pass
    
    
    if __name__ == "__main__":
        while True:
            try:
                startn = input("Starting number: ")
                startn = int(startn)
                stopn = input("End number: ")
                stopn = int(stopn)
                url_request_threaded(b, v, startn, stopn)
            except ValueError:
                print("Must be a number!")
                continue
            break