pythonc++python-3.xunit-testingboost

Can't load C++ unit test module with python 3.10 on windows (using boost 1.75.0)


I have a challenging tricky issue managing the unit tests of my C++ project in Python.

The project

The project is a big one, strongly relying on boost, with several dlls/so files and the funcitonnalities are exported to python. So I can't copy the code and it's hard to extract a minimal sample from all of that. Moreover I'm pretty sure this is more a configuration problem than a code issue. I'll give a sample of code though, concerning the critical part.

The problem

I have an unit test routine in C++ that I can use in python. It works well for both python 3.8 and 3.9. It also works with python 3.10 on Linux, but I get the following message on Windows:

<class 'ImportError'> : DLL load failed while importing UnitTestTools_4_1_0_0_beta_x64: The specified module was not found.

I checked the dependancies and they all exist and their locations are all listed in sys.path. I use sys.path.insert(0, dirPath) to add the directory containing the dll/so and pyd files. I also checked with DependenciesGui.exe

Moreover, I can use all the functionalities of my project, i.e. I can import other files and call the functions they contain. Only the unit test on windows with Python 3.10 fails to import the related module.

The configuration

C++ side, I'm using boost 1.75.0 and build the project with visual studio 2019 on Windows and gcc 9.4 on Linux. The only libraries I need to interface C++ and python are boost_python and boost_numpy.

I built boost dlls from sources (with some patches to manage newest python requirements), with dedicated Anaconda virtual environments for each version of python I need to support (3.8, 3.9 and 3.10).

Python side, when I use my libraries in python, I use the default Anaconda environment configured during the installation. I won't list all the modules installed (I can provide more if necessary) but here a comparison of python and numpy versions:

               | Windows | Linux
---------------+---------|-------
Python version |  3.9.13 | 3.9.7
Numpy version  |  1.24.1 | 1.26.3
---------------+---------|-------
Python version |  3.10.9 | 3.10.9
Numpy version  |  1.23.5 | 1.23.5

I also tried with several version of numpy for Windows/Python 3.10 (1.24.1 for instance).

I found that there was a change in the ABI between 3.9 and 3.10 but I don't think this is the reason of my problem since it works on linux.

Solving attempts

As a summary of all my attempts to solve the problem, here is what I did:

Some sample of code

Just in case it helps, I copy a part of the code concerning the unit test module export

pythonMajorVersion = sys.version_info.major
pythonMinorVersion = sys.version_info.minor
pythonVersionSubDir = "Python" + str(pythonMajorVersion) + "_" + str(pythonMinorVersion)

# add of binary directory to path
binaryDir = getBinDir() # Directory where the dlls are stored
if not binaryDir in sys.path:
    sys.path.insert(1, binaryDir)
if os.name == "nt":
    os.environ["PATH"] = os.environ["PATH"] + ";"  + binaryDir
# The python pyds are located in a specific subdirectory called "Python3_8",  "Python3_9" or  "Python3_10" depending on the python I'm using
pythonBinaryDir = os.path.join(binaryDir, pythonVersionSubDir)
if not pythonBinaryDir in sys.path:
    sys.path.insert(1, pythonBinaryDir)
if os.name == "nt":
    os.environ["PATH"] = os.environ["PATH"] + ";"  + pythonBinaryDir

 # Generate the full module name, for instance 'PyUnitTestTools_1_0_0_0_x64', 1.0.0.0 being the version number
unitTestModuleName = getLibName("PyUnitTestTools")

try:
    self.unittest = import_module(unitTestModuleName)
except:
    print("Failed to initialize unit test framework\n" + str(sys.exc_info()[0]) + " : " + str(sys.exc_info()[1]))

I'm not sure adding the libraries directories to both sys.path and os.environ["PATH"] is necessary but I tried just in case it could help.

Can't I load the unit test module in this specific configuration (Windows, Python 3.10)?


Solution

  • Thanks to procmon (as suggested by @Ahmed AEK) I found out that python 3.9 looked for dependancies in more directories than python 3.10 even though the PATH environment variable are the same for both python versions.

    My .pyd files depends on a dll located in the parent directory. This directory is in the the PATH variable environment but python 3.10 ignored it.

    I added dllDir = os.add_dll_directory(parent_path) before import_module(unitTestModuleName) and now Python 3.10 finds my dlls.