It strikes me that the run_in_executor()
method of the asyncio
library belongs to a loop
object.
In particular, what would be the difference if I chose to run a second thread alongside an async event loop the "usual" way, by import threading
, t = threading.Thread(target=...)
, t.start()
?
Perhaps the answer is by using the asyncio
module there are low level optimizations which can be made at runtime if the loop knows about additional threads?
You can always start another thread manually, but then you are responsible for giving it work, e.g. using a queue. In Python 3 concurrent.futures
provide a convenient API for offloading tasks to a thread pool, which it calls executor. Its submit
method takes a function, gives it to a thread in the pool to run it, and immediately returns a handle that will provide the result (or propagate an exception) when it is ready.
run_in_executor
delivers the same convenience to asyncio. Remember that you're not supposed to run any blocking code inside asyncio - for example time.sleep()
is forbidden, because it blocks the whole event loop. run_in_executor
allows you to side-step that rule. For example:
async def sleep_test():
loop = asyncio.get_event_loop()
print('going to sleep')
await loop.run_in_executor(None, time.sleep, 5)
#time.sleep(5)
print('waking up')
async def parallel():
# run two sleep_tests in parallel and wait until both finish
await asyncio.gather(sleep_test(), sleep_test())
asyncio.run(parallel())
Running this code shows that both instances of the coroutine sleep in parallel. If we called time.sleep()
directly, they'd sleep in series despite the call to asyncio.gather()
because time.sleep()
would block the event loop.
This example is of course silly because there is asyncio.sleep()
that suspends a coroutine without taking up a slot in the thread pool, but it shows the basic idea. Realistic use cases for run_in_executor
include: