[SOLVED] asyncio - wait for each task in a dynamic list to finish or be cancelled

asyncio - wait for each task in a dynamic list to finish or be cancelled

I have a list of asyncio tasks: [task1, task2, task3]. And each task can append more tasks to this list, e.g., task3 might append task4.

How can I wait for all tasks in this dynamic list to finish or be cancelled?

I tried await asyncio.gather(*[task1, task2, task3]), but it doesn't wait for tasks created by task1, task2 or task3.

Solution

You can use a loop to wait for as long as there are unfinished tasks:

# create a list to hold the tasks
task_list = []

# you can pass task_list to coroutines that need to submit
# additional tasks
task_list.extend([task1, coro1(arg1, task_list), coro2(arg2, task_list)])

while task_list:
    old_cnt = len(task_list)
    await asyncio.gather(*task_list)
    # we've finished the old set of tasks, but keep looping as long
    # as some coroutines keep adding new ones
    del task_list[:old_cnt]

Contents of task_list don't need to be just actual asyncio.Task instances, the list can also contain coroutine objects or anything else that is awaitable. You can pass task_list to the coroutines that need to append more tasks to the list. Once no new tasks have been added, the task list will remain empty after the call to gather, and the loop will terminate.