I'm working on a Python project with a setup.py that has something like this1:
setup(
cmdclass={"build_ext": my_build_ext},
ext_modules=[
Extension("A", ["a.c", "common.c"]),
Extension("B", ["b.c", "common.c"])
]
)
I'm running into a problem when building the modules in parallel where it seems like one module tries to read common.o
/common.obj
while another is compiling it, and it fails. Is there some way to get setuptools to compile the C files for each module into their own build directories so that they aren't overwriting each other?
I found a potential solution by overriding build_extension()
in a custom build_ext
class:
import copy, os
from setuptools import Extension, setup
from setuptools.command.build_ext import build_ext
class my_build_ext(build_ext):
def build_extension(self, ext):
# Append the extension name to the temp build directory
# so that each module builds to its own directory.
# We need to make a (shallow) copy of 'self' here
# so that we don't overwrite this value when running in parallel.
self_copy = copy.copy(self)
self_copy.build_temp = os.path.join(self.build_temp, ext.name)
build_ext.build_extension(self_copy, ext)
setup(
cmdclass={"build_ext": my_build_ext},
ext_modules=[
Extension("A", ["a.c", "common.c"]),
Extension("B", ["b.c", "common.c"])
]
)
I've also since been told of the (currently undocumented) libraries
parameter for setup()
:
from setuptools import Extension, setup
setup(
libraries=[
("common", {"sources": ["common.c"]}),
],
ext_modules=[
Extension("A", sources=["a.c"], libraries=["common"]),
Extension("B", sources=["b.c"], libraries=["common"]),
],
)
Both solutions worked for me, but in slightly different ways. The first solution recompiles the code for each module, which allows you to specify different parameters to use for each module (ex. different defs
). The second solution only has to compile to code once and it will reuse that for every module.