pythonasynchronousmultiprocessingpython-asyncioprocess-pool

How to run multiple asyncio loops inside syncrhonous sub-processes inside a main asyncio loop?


I have a main function which I run with asyncio and then inside it I used event_loop.run_in_executor() to run mutliple processes of a blocking function.

What I wish to do is inside each of these processes to run a new asyncio loop for each one in order to execute async code.

So I have a main async function where I create multiple processes and I want to create a new asyncio loop in each of them. How can this be done?


Solution

  • You can just call asyncio.run within your subprocess with a coroutine with the async work you want to do. A minimal example that spins up 5 processes each with their own event loops:

    import asyncio
    from concurrent.futures import ProcessPoolExecutor
    
    
    def run_loop_in_process():
        async def subprocess_async_work():
            await asyncio.sleep(5) #whatever async code you need
        asyncio.run(subprocess_async_work())
    
    async def main():
        loop = asyncio.get_running_loop()
        pool = ProcessPoolExecutor()
        tasks = [loop.run_in_executor(pool, run_loop_in_process) for _ in range(5)]
        await asyncio.gather(*tasks)
    
    if __name__ == "__main__":
        asyncio.run(main())