[SOLVED] Effect of using sys.path.insert(0, path) and sys.path.append(path) when loading modules

Effect of using sys.path.insert(0, path) and sys.path.append(path) when loading modules

I was recently having a problem with a python ImportError, where the module was found when running on my local computer but not found on the CI server. I solved this problem by swapping sys.path.append(path) in my script with sys.path.insert(0, path) where path is the string module location.

Since this is my module and not an installed package (related question), why does the order of paths fix this problem?

Solution

Because Python checks in the directories in sequential order, starting at the first directory in the sys.path list, until it finds the .py file it was looking for.

Normally, the current directory or the directory of the script is the first element in the list, unless you modify it (like you did). From documentation:

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH.

So, most probably, you had a .py file with the same name as the module you were trying to import from, in the current directory (where the script was being run from).

Also, something to note about ImportErrors - Let's say the import error says: ImportError: No module named main. This doesn't mean main.py is overwritten - if it was overwritten it would still exist and we would not have issues trying to read it. It's some module above this that got overwritten with a .py or some other file.

For example, consider a directory structure that looks like this:

 - test
    - shared
       - __init__.py
       - phtest.py
    - testmain.py

Now from testmain.py, I call from shared import phtest, which works fine.

Now let's say I introduce a shared.py in test directory:

 - test
    - shared
       - __init__.py
       - phtest.py
    - testmain.py 
    - shared.py

Now when I try to do from shared import phtest from testmain.py, I get the error:

ImportError: cannot import name 'phtest'

As you can see above, the file that introduced the issue is shared.py, not phtest.py.