I want to lock a directory and its contents, to allow only one process writing in it.
To do that, I'm using the Portalocker library, and a file called .lock
from pathlib import Path
import os
import portalocker as pl
DIRECTORY = Path('./protected')
p_lock = DIRECTORY / '.lock'
with pl.Lock(p_lock, 'w', timeout=10) as lock:
# Writing pid to lock
print(os.getpid(), file=lock)
lock.flush()
# do things that write to some files
# inside DIRECTORY
# Remove lock file
p_lock.unlink()
As long as all the processes execute the same code, no two process will be in the critical section. This wasn't causing any problems, but now I want to make a reader process.
from pathlib import Path
import os
import portalocker as pl
DIRECTORY = Path('./protected')
p_lock = DIRECTORY / '.lock'
with pl.Lock(p_lock, 'r', timeout=10):
# do things
This throws a FileNotFoundError: [Errno 2] No such file or directory: './protected/.lock', because the file is removed when the writer finishes.
How can I make sure that either one process is writing, or multiple processes is reading to/from a directory?
Note: There is one writer process, and multiple readers (threads spawned from the same process, but not the same as the writer)
This is an issue with how portalocker deals with files - it tries to open them, which works when opening non-existing file in 'w' mode, and fails in 'r' mode.
The solution in your case is to manually create the file (and never remove it, it's a bad idea).
Writer:
with pl.Lock(p_lock, 'w'):
# do things
# don't remove the p_lock file
Reader:
p_lock.touch(exist_ok=True)
with pl.Lock(p_lock, 'r', flags=pl.LockFlags.SHARED | pl.LockFlags.NON_BLOCKING):
# do things
# don't remove the p_lock file
(and don't bother writing PID into the file, unless it's for your own debugging purpuses)