Cannot Get Import to Work with Subpackages

The following structure (in Python 3.7) is not allowing me to import class A in module b:

package:
    package:
        __init__.py
        a:
            __init__.py
            a.py
        b:
            __init__.py
            b.py

The top-level __init__.py is blank. Here are the remaining files:

# package/package/a/__init__.py

from .a import A

# package/package/a/a.py

class A:

    def __init__(self):
        pass

# package/package/b/__init__.py

from .b import B

# package/package/b/b.py

from package.a.a import A

class B:

    def __init__(self):
        pass

Without doing anything else, on Windows, if I try to run b.py (from within the b folder), I get the following error:

ModuleNotFoundError: No module named 'package.a'

If I add a main.py at the top level:

package:
    package:
        __init__.py
        main.py
        a:
            __init__.py
            a.py
        b:
            __init__.py
            b.py

containing

# package/package/main.py

import a
import b

and run main.py (from within package/package), I get the same error.

If I change b.py to

# package/package/b/b.py

from ..a.a import A

class B:

    def __init__(self):
        pass

and run b.py (from within the b folder) or main.py (from within package/package), I get the error that

ValueError: attempted relative import beyond top-level package

The python docs make it seem like I should be able to do this though!

Can someone please explain why I am getting these errors? I've seen a couple similar posts to this, but they haven't fixed my problem:

Solution

Whatever module is being run by Python is called top-level.

In your shell, when you run > py main.py ($ python3 main.py on Linux), the file main.py is top-level and is called the top-level module.
In the interpreter, the interpreter itself is always top-level, and is called the top-level environment (for proof, type >>> __name__ into the interpreter and it will return '__main__')

Unfortunately (IMO), the term "top-level" is not well-defined in the python docs as it is used in several different contexts. Regardless, it is important to understand that Python always renames __name__ of the top-level entity to '__main__'.

PEP 328 (explained in this SO post) states

relative imports use the module's __name__ attribute to determine its position in the package hierarchy.

Since Python renames the __name__ of the top-level module to '__main__', the top-level module has no package information because it has no dots in its name.

Hence, top-level modules are not considered part of any package (even though they very well may be!). In other words, for packages imported from the current directory, '__main__' determines what is top-level. Packages at the same level as '__main__' (a and b in my example) are top-level packages.

Confusingly, the python docs and PEP 328 give a misleading example. The "correct usages" shown for relative imports are only valid in a specific context.

Recall that import searches paths listed in sys.path to find packages and modules to import. The current directory is in sys.path, along with the paths to builtin packages (like math and os) and installed packages (i.e. pip installed package). Python does not rename the __name__ of non-top-level packages.

Therefore, the python docs and PEP 328 examples are valid only for packages and modules NOT in the top-level directory.

What I should have written was from a.a import A:

# package/package/b/b.py

from a.a import A

class B:

    def __init__(self):
        pass

Since package is above the top-level module, trying to do an absolute import (like from package.a.a import a) results in an ImportError even though main.py is inside of the package package.

That being said, if you go to PyPI and GitHub and look at released packages, you will find they have absolute imports like import package.a.a! In fact, if I were to publish my package and leave the import as from a.a import A, end users would get an ImportError because they installed package package and don't have a package a! Furthermore, in the current configuration, I'm unable to test with unittest or pytest that users can import and use my package as expected because I cannot do from package.a.a import A myself!

So the question becomes how do you write and test your own custom packages?

The trick is that these packages were written in development mode. To install your package in development mode, run > pip install -e . from the top-level directory (assuming you have a setup.py written; see the NOTE below).

When this is done, python treats your package like a typical library package (i.e. a pip installed package), so python does not change its __name__ to __main__. Thus, you can

import it with absolute imports
test and use your package like an end user would, and
any edits you make to it take effect immediately when run without requiring you re-pip install it, just like packages in the top-level directory do

This key difference between developing packages vs. standalone programs is a huge source of confusion and frustration for most first-time developers (myself included) and is very important to keep in mind. I hope this answer provides clarification for others and may be added to documentation in the future. Please let me know in the comments below if this helped you.

NOTE: pip install -e ., where -e stands for "editable", puts a link (a *.pth file) in your python installation folder so that the package is treated as an installed package, but also that any changes you write in it will take effect immediately (see the Python Packaging Tutorial). Hence, you can use this to develop your own packages or to install and edit third-part packages to your needs. This requires you create a setup.py, but all your test code, client code, etc., will be able to import your package the usual way (i.e. you can treat your package like any other pip installed package). You can achieve the same effect with poetry and flit by configuring your pyproject.toml file.

Here are some additional useful references: