I have a django project running inside docker, and the service is up with the command of python manage.py runserver
, with file autoreload open, and using threadings.
My code invokes shutil.make_archive()
which will then invoke os.getcwd()
, and from time to time, os.getcwd()
will raise FileNotFoundError
, by searching more information, now I realized that the error might be caused by the path is no more there because the path is deleted by somewhere else after I enter the path.
The error only raised sometimes and I couldn't find a stable way to reproduce it, once it happens, I can fix the problem by making any code changes to trigger file autoreload or restart the docker service, looks like make sure the process restarts will do the help, otherwise the error keeps raising.
When things going properly, os.getcwd()
will return my django project root dir path.
And I'm 100% sure my code does nothing related with dir manipulation like os.chdir()
.
So in a nutshell, the django server works fine, suddenly os.getcwd()
starts to raise error until I restart the process.
My question is, what could be the root cause? is it possible that python manage.py runserver
might cause any issue?
Python: v3.8.6
Django: v3.2
Docker: v20.10.12
Turns out that it has nothing to do with dev server or docker.
It's because shutil.make_archive
is not thread safe.
What shutil.make_archive
does is:
pushd .; popd;
)I'm calling shutil.make_archive
from multiple threads simultaneously, and also rmdir those temp dirs, and since os.chdir
takes effect process-wise, so things happen in the order of:
and in the final step FileNotFoundError
raised thanks to the race condition.
For those who want to dig more about shutil.make_archive
's thread-safety, there is a thread discussed about it.