I've been using the python requests library for some time, and recently had a need to make a request asynchronously, meaning I would like to send off the HTTP request, have my main thread continue to execute, and have a callback called when the request returns.
Naturally, I was lead to the grequests library (https://github.com/kennethreitz/grequests), but i'm confused about the behavior. For example:
import grequests
def print_res(res):
from pprint import pprint
pprint (vars(res))
req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res))
res = grequests.map([req])
for i in range(10):
print i
The above code will produce the following output:
<...large HTTP response output...>
0
1
2
3
4
5
6
7
8
9
The grequests.map() call obviously blocks until the HTTP response is available. It seems likely I misunderstood the 'asynchronous' behavior here, and the grequests library is just for performing multiple HTTP requests concurrently and sending all responses to a single callback. Is this accurate?
.map() is meant to run retrieval of several URLs in parallel, and will indeed wait for these tasks to complete (gevent.joinall(jobs)) is called).
Use .send() instead to spawn jobs, using a Pool instance:
req = grequests.get('http://www.codehenge.net/blog', hooks=dict(response=print_res))
job = grequests.send(req, grequests.Pool(1))
for i in range(10):
print i
Without the pool the .send() call will block still, but only for the gevent.spawn() call it executes.