I am using the grequests library to pass ~250000 urls to get data from an api.
The API has a limit of 100 calls per second.
How do I throttle the grequests to only pass in 100 urls per second? I increased the size parameter from 5 to 100. Not sure what this does but still running to error 'Max retries exceeded'.
Here is my code so far:
import grequests
lst = ['url.com','url2.com']
class Test:
def __init__(self):
self.urls = lst
def exception(self, request, exception):
print ("Problem: {}: {}".format(request.url, exception))
def async(self):
return grequests.map((grequests.get(u) for u in self.urls), exception_handler=self.exception, size=100)
def collate_responses(self, results):
return [x.text for x in results]
test = Test()
#here we collect the results returned by the async function
results = test.async()
response_text = test.collate_responses(results)
Grequests seems to make 100 requests and then without any waiting makes another 100 requests and so on. There is no time defined between these requests. Here is a similar problem described with a solution: Limiting/throttling the rate of HTTP requests in GRequests