Building a Python extension with a local C++ library dependency using setuptools and build

I have a library written in C++ for which I would like to make a Python wrapper. The directory structure of my repository is similar to the following:

.
├── include # C++ library headers
│   └── …
├── Makefile
├── python # Python package
│   ├── wrapper-module # C++ and Python source files
│   │   └── …
│   ├── pyproject.toml
│   ├── setup.py
│   └── …
├── src # C++ library source files
│   └── …
└── tests # C++ library tests
    └── …

I use GNU make for building the C++ library and use Python’s C API directly to implement the wrapper. (I.e. no Boost.Python, Cython etc.)

Building the Python package has been unsuccessful so far. Since I use the C API, I chose setuptools 80.9.0 for building the module. When running python -m build, the relevant files are copied to a separate location, even if I use --no-isolation. So far I have been unable to make the C++ library available in that location or determine their relative path with respect to that location. (I would like to avoid specifying its absolute path or installing it to e.g. /usr.) The source files for the wrapper module (or at least most of them) are copied as expected.

So far I have attempted to solve the issue as follows:

• Checking the documentation. What I have found so far has had to do with Python dependencies, not “local” ones.
• Copying the C++ library build products to the Python directory with a build phase in the Makefile. In this case the build products are not copied to the separate build location. I have not found a configuration option to specify the directory in question as a build dependency.
• Adding a custom build phase in setup.py. In this case I have not found a function or property to retrieve the original path in order to copy the required files as part of the build process.
• Writing a custom build backend based on setuptools.build_meta. In this case I can determine the original path using os.getcwd(). However, the module that contains the custom backend is not copied to the separate build location even though I have set build-backend and backend-path in pyproject.toml. (In any case I probably would need to store the original path somehow, to which I have not paid much attention.)

Any suggestions for making setuptools aware of the location of the built C++ library and the associated header files are appreciated.

I finally came up with a solution, although there are some reasons to consider other options: According to PEP 517, the source tree should be self-contained, so one could argue that the source code of the C++ library should be included in it. I did not want to change the directory structure or add another repository for the Python package, though. Apprently using a build backend such as scikit-build would have made it possible (easier?) to use CMake as part of the build system instead of trying to build everything with setuptools, but I did not want to switch to that either.

The basic idea is to:

• (Ab)use Python’s build module’s configuration options to pass the current directory to the build system with a command similar to python -m build -C project-root=`pwd`. Running the command can be made easier by writing it to a Makefile.
• Write a build backend based on setuptools.build_meta that handles the configuration option.
• Write a build command based on setuptools.command.build_ext.build_ext that modifies relative header and library paths.

Here is the updated directory structure.

python
├── Makefile
├── pyproject.toml
├── README.md
├── my-python-package
│   ├── __init__.py
│   ├── _custom_build.py # Build backend and command
│   ├── py.typed
│   └── my-python-module.pyi
└── src
    └── my-python-module.cc

Here are the relevant parts of pyproject.toml. I used relative paths for include-dirs and library-dirs under [[tool.setuptools.ext-modules]].

[build-system]
requires = ["setuptools >=80.9"] # An older version might have sufficed.
build-backend = "_custom_build"
backend-path = ["my-python-package"]

[tool.setuptools]
packages = ["my-python-package"]

[tool.setuptools.cmdclass]
build_ext = "_custom_build.build_ext"

Here is _custom_build.py.

from setuptools.build_meta import *
from setuptools.build_meta import build_wheel as build_wheel_
from setuptools.command.build_ext import build_ext as build_ext_
import typing


project_root: str = ""


def fix_path(path: str) -> str:
    if path.startswith("/"):
        return path
    return f"{project_root}/{path}"


def build_wheel(wheel_directory, config_settings = None, metadata_directory = None):
    global project_root
    if config_settings:
        project_root = config_settings.get("project-root", "")
    return build_wheel_(wheel_directory, config_settings, metadata_directory)


class build_ext(build_ext_):
    @typing.override
    def initialize_options(self):
        super().initialize_options()
        # Add the include and library paths outside the build directory,
        # since the library is built separately.
        for ext in self.distribution.ext_modules:
            ext.include_dirs = list(map(fix_path, ext.include_dirs))
            ext.library_dirs = list(map(fix_path, ext.library_dirs))