pythonc++importpyd

Python import mechanism when using a .pyd file

I consume a Python package which currently uses C++ under the hood and which was only build for Python 3.8.

The package content looks like this:

my_package/
|
|- __init__.py
|- additional_code.py
|- my_lib.dll
|- wrapper.cp38-win_amd64.pyd

The __init__.py exposes everything from the .dll:

from .wrapper import *
...

My main.py consumes this as following:

# main.py

import my_package as pkg
pkg.my_method()

Currently this works only for Python 3.8 but fails for Python > 3.8.

How exactly does Python's import mechanism works here to get it also running for Python 3.9?

Is it sufficient to only add a wrapper.cp39-win_amd64.pyd (i.e., a wrapper build from Python 3.9) or do I need to write an own "import helper" which checks the current Python interpreter version?

I have found Python's import documentation, but I did not really find my exact use case. Maybe, someone can me point to the right direction.

Solution

  • just re-compile the file using python 3.9 environment and headers, most of python's C API is pretty stable across minor versions (3.8 -> 3.9) so most code just needs to be recompiled to work for a different interpreter version.

    there are two parts that contribute to a .pyd compiled for one version of python not working for another version until re-compiled.

    1. when you use import .wrapper, python will look for a .pyd file whose name contains the version number that matches the running interpreter version, so it will be searching for wrapper.cp39-win_amd64.pyd then it will try to load it with LoadLibrary, which leads to the second problem
    2. when you compile a .pyd against python 3.8 it will link to python38.dll, so even if you renamed the already existing .pyd, it will probably fail to load. (And if it did load it will crash)

    if you want your "wrapper" to work across multiple versions of python without compilation you should compile your library as a .dll with a C interface and load it using ctypes, and not depend on python C API inside your dll.