I would like to perform cython
files compilation in parallel.
So, I take a look at Cython.Build
source file, and find the following signature for cythonize
function:
def cythonize(module_list, exclude=None, nthreads=0, aliases=None,
quiet=False, force=False, language=None,
exclude_failures=False, **options):
And the following comment about cythonize nthreads
option:
"For parallel compilation, set the 'nthreads' option to the number of
concurrent builds."
So I tried to use this option in my setup.py
file, like that:
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils.extension import Extension
EXTENSIONS = [Extension(...)
...
Extension(...)]
setup(name='...',
...
ext_modules=cythonize(EXTENSIONS, nthreads=8),
...)
But my .pyx
files are still compiled sequentially using 1 thread.
I do not understand what I am doing wrong here and how to use nthreads
option to perform cythonize
compilation in parallel ?
I finally found a solution to compile my cython file in parallel:
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils.extension import Extension
NB_COMPILE_JOBS = 8
EXTENSIONS = [Extension(...)
...
Extension(...)]
def setup_given_extensions(extensions):
setup(name='...',
...
ext_modules=cythonize(extensions),
...)
def setup_extensions_in_sequential():
setup_given_extensions(EXTENSIONS)
def setup_extensions_in_parallel():
cythonize(EXTENSIONS, nthreads=NB_COMPILE_JOBS)
pool = multiprocessing.Pool(processes=NB_COMPILE_JOBS)
pool.map(setup_given_extensions, EXTENSIONS)
pool.close()
pool.join()
if "build_ext" in sys.argv:
setup_extensions_in_parallel()
else:
setup_extensions_in_sequential()