Python can't see a new module in a directory it already imported from

In this test, I create a new directory (deleting it if it already exists), add it to sys.path, add two simple Python files, and try to import them. I'm using Python 3.11.6 on Ubuntu 23.10.

import sys
from pathlib import Path
from shutil import rmtree

d = Path('/tmp/fleen')
try:
    rmtree(d)
except FileNotFoundError:
    pass
d.mkdir()
sys.path = [str(d)] + sys.path

(d / 'module1.py').write_text('x = 1')
import module1
assert module1.x == 1

(d / 'module2.py').write_text('x = 2')
print('file contains:', repr((d / 'module2.py').read_text()))
import module2
assert module2.x == 2

This prints file contains: 'x = 2', as expected, but then raises

Traceback (most recent call last):
  File "example.py", line 16, in <module>
    import module2
ModuleNotFoundError: No module named 'module2'

What's going on? Why can't import see the file? Is there some sort of cache of each directory in sys.path that I need to clear?

Edited to add: Unpredictably, this sometimes works, as if there's a race condition of some kind. This is mysterious because all the IO here is supposed to flush all buffers before going on to the next statement.

Solution

Some importers cache the modules they find. You have to invalidate the cache when another module is added. From importlib.import_module

If you are dynamically importing a module that was created since the interpreter began execution (e.g., created a Python source file), you may need to call invalidate_caches() in order for the new module to be noticed by the import system.

By adding a line to invalidate importer caches after writing the second file, the problem is solved

import sys
from pathlib import Path
from shutil import rmtree
import importlib

d = Path('/tmp/fleen')
try:
    rmtree(d)
except FileNotFoundError:
    pass
d.mkdir()
#sys.path = [str(d)] + sys.path
sys.path.insert(0, str(d))

(d / 'module1.py').write_text('x = 1')
import module1
assert module1.x == 1

(d / 'module2.py').write_text('x = 2')
importlib.invalidate_caches()                 # <== invalidate
print('file contains:', repr((d / 'module2.py').read_text()))
import module2
assert module2.x == 2

This wasn't needed for the first file because /tmp/fleen hadn't been visited by the importer before the import.