pythonsetuptoolspython-c-api

Python setuptools multiple extension modules with shared C source code building in parallel


I'm working on a Python project with a setup.py that has something like this1:

setup(
    cmdclass={"build_ext": my_build_ext},
    ext_modules=[
        Extension("A", ["a.c", "common.c"]),
        Extension("B", ["b.c", "common.c"])
    ]
)

I'm running into a problem when building the modules in parallel where it seems like one module tries to read common.o/common.obj while another is compiling it, and it fails. Is there some way to get setuptools to compile the C files for each module into their own build directories so that they aren't overwriting each other?

 

  1. The actual project is more complicated with more modules and source files.

Solution

  • I found a potential solution by overriding build_extension() in a custom build_ext class:

    import copy, os
    from setuptools import Extension, setup
    from setuptools.command.build_ext import build_ext
    class my_build_ext(build_ext):
        def build_extension(self, ext):
            # Append the extension name to the temp build directory
            # so that each module builds to its own directory.
            # We need to make a (shallow) copy of 'self' here
            # so that we don't overwrite this value when running in parallel.
            self_copy = copy.copy(self)
            self_copy.build_temp = os.path.join(self.build_temp, ext.name)
            build_ext.build_extension(self_copy, ext)
    setup(
        cmdclass={"build_ext": my_build_ext},
        ext_modules=[
            Extension("A", ["a.c", "common.c"]),
            Extension("B", ["b.c", "common.c"])
        ]
    )
    

    I've also since been told of the (currently undocumented) libraries parameter for setup():

    from setuptools import Extension, setup
    setup(
        libraries=[
            ("common", {"sources": ["common.c"]}),
        ],
        ext_modules=[
            Extension("A", sources=["a.c"], libraries=["common"]),
            Extension("B", sources=["b.c"], libraries=["common"]),
        ],
    )
    

    Both solutions worked for me, but in slightly different ways. The first solution recompiles the code for each module, which allows you to specify different parameters to use for each module (ex. different defs). The second solution only has to compile to code once and it will reuse that for every module.