I want to do parallel http request tasks in asyncio
, but I find that python-requests
would block the event loop of asyncio
. I've found aiohttp but it couldn't provide the service of http request using a http proxy.
So I want to know if there's a way to do asynchronous http requests with the help of asyncio
.
To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example:
import asyncio
import requests
@asyncio.coroutine
def main():
loop = asyncio.get_event_loop()
future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
response1 = yield from future1
response2 = yield from future2
print(response1.text)
print(response2.text)
asyncio.run(main())
This will get both responses in parallel.
With python 3.5 you can use the new await
/async
syntax:
import asyncio
import requests
async def main():
loop = asyncio.get_event_loop()
response1 = await loop.run_in_executor(None, requests.get, 'http://www.google.com')
response2 = await loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
print(response1.text)
print(response2.text)
asyncio.run(main())
See PEP0492 for more.