pythonrefactoringpython-typingcircular-dependencypython3.15

How do I resolve this circular import?


Implementing a virtual file system I encounter a circular import problem.

common.py:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from .directory import Directory
    from .archive import Archive

class VFile:
  def __init__(self, data: bytes, parent: Directory | Archive ...):
    ...

archive.py:

from .util import guess_type

class Archive(VFile):
  def _read(self):
    for m in metadata:
      data = ...
      FileType = guess_type(data)
      self.files.append(FileType(data, self))
    ...

directory.py:

from .util import guess_type

class Directory(VFile):
  def _parse_btree(self):
    FileType = guess_type(data)
    self.files.append(FileType(data, self))
    ...

util.py:

from .common import VFile
from .archive import Archive
from .directory import Directory

def guess_type(data: bytes) -> type[VFile]:
  if is_archive(data):
    return Archive
  if is_dir(data):
    return Directory
  ...
  return VFile

TYPE_CHECKING solves part of the problem, but I'm unable to resolve the rest without concatenating all files into one. How do I refactor this?

Typehint circular is not relevant here. Real circular is Archive calling guess_type() which references Archive and Directory also calls guess_type() which references Directory.


Solution

  • Archive and Directory will need to call guess_type only when they are called - and the same, guess type will also only need those when called, not when importing the module.

    The one occasion you'd need those at import time, would be if any of the three objects (The Directory and Archive classes, or the guess_type function - I am naming all three as "objects" in this text) would need to be in the annotations of each other - for this the if TYPE_CHECKING guard will work.

    For the actual usage in runtime you will have two easy solutions: Just import the modules as the module name, and use . attribute access to get to either object.

    So, this would work:

    util.py

    import typing
    
    
    from . import archive
    from . import directory
    from . import common
    
    # from Python 3.14 on, you can use the same patter for VFile as well,
    # as annotations are now lazily evaluated! (check PEP649 - https://peps.python.org/pep-0649/ )
    if typing.TYPE_CHECKING:
          from .common import VFile
    
    
    def guess_type(data: bytes) -> type[VFile]:
      if is_archive(data):
        return archive.Archive
      if is_dir(data):
        return directory.Directory
      ...
      return common.VFile
    
    
    

    archive.py, directory.py

    from . import util
    
    FileType = util.guess_type(data)
    

    Another simple approach is to add the import statements inside the function itself:

    
    # resolve VFile import as above
    ...
    
    def guess_type(data: bytes) -> type[VFile]:
      from .archive import Archive
      from .directory import Directory
      from .common import VFile
      if is_archive(data):
        return Archive
      if is_dir(data):
        return Directory
      ...
      return VFile
    
    
    

    The drawback of this, is that even though Python won't perform the "full import operation", of reading the file, etc... (each module is actually read from disk once per process, and this is a basic caching in Python since the very first versions), the import statement itself, even though it will be performing just a few checks and creating the aliases here, is a slow operation compared to assignments and dot access. (~10 times slower) - so, if this function is called in a tight loop, you might want to optimize that, by using some global names and performing the inside-the-function import guarded by an if.

    import typing
    
    if typing.TYPE_CHECKING:
    
        from .common import VFile
        from .archive import Archive
        from .directory import Directory
    
        Archive: Archive
        Directory: Directory
    
    def guess_type(data: bytes) -> type[VFile]:
        global Archive, Directory
        if "Archive" not in globals():
            from .archive import Archive
            from .directory import Directory 
            from .common import VFile
            
        if is_archive(data):
            return Archive
        if is_dir(data):
            return Directory
        ...
        return VFile       
    
    

    In this answer I just break the cycle in the your utils file, but you can use the same approach on any of the other files a well - even having some redundancy if you want.

    And last, but not least, this is a known problem, and Python language, developed as it is by the community, without pushes from any corporation, has approved PEP-810 featuring a dedicated built-in mechanism for lazy imports - but that will only be available with Python 3.15.0, scheduled for release on October 2026.

    When you run a post PEP-810 Python, this should work out of the box (you could download an alpha version now, but not for production):

    lazy from .common import VFile
    lazy from .archive import Archive
    lazy from .directory import Directory
    
    def guess_type(data: bytes) -> type[VFile]:
      if is_archive(data):
        return Archive
      if is_dir(data):
        return Directory
      ...
      return VFile