For some reason, this code doesn't work on Ubuntu 20, Python 3.8.10, unless the .gather line is commented out.
It works on Ubuntu 24, Python 3.12.3 and Windows 11, Python 3.13.9.
It doesn't work on Python 3.8/3.9 on the same Ubuntu 24 machine. It seems to work from Python 3.10 onwards, so the issue appears to be one with the Python version.
This code is a minimum working example of the issue that I'm facing. Assume that the locks are actually necessary.
import asyncio
import time
lock = asyncio.Lock()
async def action():
print("action")
return
async def task1():
while True:
print("1 waiting")
await lock.acquire()
print("1 running")
await asyncio.gather(*[action() for _ in range(1)])
lock.release()
print("1 unlocked")
print("A", time.time())
await asyncio.sleep(1)
print("B", time.time())
async def task2():
while True:
print("2 waiting")
await lock.acquire()
print("2 running")
lock.release()
print("2 unlocked")
await asyncio.sleep(1)
async def main():
task1_ = asyncio.create_task(task1())
task2_ = asyncio.create_task(task2())
while True: await asyncio.sleep(float('inf'))
if __name__ == "__main__":
asyncio.run(main())
The output on one of the Python versions that works is:
$ python3 test.py
1 waiting
1 running
2 waiting
action
1 unlocked
A 1763606395.8323498
2 running
2 unlocked
B 1763606396.843853
1 waiting
1 running
2 waiting
action
1 unlocked
A 1763606396.8451693
2 running
2 unlocked
B 1763606397.8484037
Notice that task2 gets to run between A and B (while task1 is asleep).
The output of the non-working versions is:
$ python3 test.py
1 waiting
1 running
2 waiting
action
1 unlocked
A 1763606434.4998655
B 1763606435.5010738
1 waiting
1 running
action
1 unlocked
A 1763606435.5020432
B 1763606436.5031264
1 waiting
1 running
action
1 unlocked
A 1763606436.5039277
B 1763606437.5050313
As you can see, task2 doesn't run between A and B.
This issue occurs regardless of using await lock.acquire() or async with lock:.
If I comment out the .gather line, it works on all versions.
However, I need to run multiple tasks at the same time, so I can't simply comment that out.
I haven't been able to find anything in the docs that suggests an issue with gather or locks in older versions of asyncio.
So what's going on here, and how do I fix it?
The fact that the code works as expected on Python >=3.10 is a side effect of removing the loop parameter.
Prior to Python 3.10, asyncio.Lock was bound to the current event loop at initialization time[1]. That is, in your case, outside of asyncio.run(). As a result, on Python <3.10, future objects are created on behalf of the wrong event loop[2], and attempting to use them results in a RuntimeError. Because of this, your second task failed (since the lock was already held by the first task), and the only active task was the first one.
import asyncio
lock = asyncio.Lock()
async def test():
await lock.acquire() # actually non-blocking call
await asyncio.wait_for(lock.acquire(), 1e-3) # blocking call (deadlock)
# `RuntimeError` on Python <3.10
# `TimeoutError` on Python >=3.10
asyncio.run(test())
Starting with Python 3.10, asyncio.Lock binds to the current event loop at the first blocking call[3], and therefore it creates future objects on behalf of the right event loop. If you want to support Python <3.10, do not create asyncio synchronization primitives at the module level or anywhere else outside the event loop.
The reason why you did not find the real cause of the problem is that you use asyncio.create_task() to create new tasks and do not handle their failures in any way. Exceptions from tasks that no one is waiting for are only logged when they are deleted[4][5]. And since you refer to them in the main() function, they will not be deleted until it finishes its execution (for example, by Control-C).
As for removing the line with asyncio.gather(), this simply means that no context switching occurs between acquiring the lock and releasing it, and thus no task ever sees the lock in the locked state. This is because asyncio implements cooperative multitasking, and you can read more about this either in the documentation or in the answers to a related question.
If you want to work safely with tasks, consider using asyncio.TaskGroup. There is also a backport to Python <3.11.
A quick way to verify the cause is to add the following lines to the beginning of the main() function:
global lock
lock = asyncio.Lock()
With this change, your code works as expected on Python <3.10.