pythonpython-requestspython-3.4aiohttp

How could I use requests in asyncio?


I want to do parallel http request tasks in asyncio, but I find that python-requests would block the event loop of asyncio. I've found aiohttp but it couldn't provide the service of http request using a http proxy.

So I want to know if there's a way to do asynchronous http requests with the help of asyncio.


Solution

  • To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example:

    import asyncio
    import requests
    
    @asyncio.coroutine
    def main():
        loop = asyncio.get_event_loop()
        future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
        future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
        response1 = yield from future1
        response2 = yield from future2
        print(response1.text)
        print(response2.text)
    
    asyncio.run(main())
    

    This will get both responses in parallel.

    With python 3.5 you can use the new await/async syntax:

    import asyncio
    import requests
    
    async def main():
        loop = asyncio.get_event_loop()
        response1 = await loop.run_in_executor(None, requests.get, 'http://www.google.com')
        response2 = await loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
        print(response1.text)
        print(response2.text)
    
    asyncio.run(main())
    

    See PEP0492 for more.