I have a list of ~300K API URLs that I want to call and get data from:
lst = ['url.com','url2.com']
If I subset my list down to say 5 urls grequest
handles the request perfectly. However when I pass in the full ~300K URLs I get error:
Problem: url.Iam.passing.in: HTTPSConnectionPool(host='url', port=xxx): Max retries exceeded with url: url.Iam.passing.in (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x552b17550>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))
Traceback (most recent call last):
Code so far to make the asynchronous calls:
class Test:
def __init__(self):
self.urls = lst
def exception(self, request, exception):
print ("Problem: {}: {}".format(request.url, exception))
def async(self):
return grequests.map((grequests.get(u, stream=False) for u in self.urls), exception_handler=self.exception, size=5)
def collate_responses(self, results):
return [x.text for x in results]
test = Test()
#here we collect the results returned by the async function
results = test.async()
response_text = test.collate_responses(results)
I am not sure what I am doing wrong, when I have even passed stream=False
.
Is there anyway I can pass my list in batches?
Try something on these lines:
def async(x):
#.....do something here.....#
#return grequests.map((grequests.get(x, stream=False)), exception_handler=self.exception, size=5)
for url in url_list:
result = async(url)
time.sleep(5) #This will add a 5 second delay