pythonpython-2.7cythoncythonize

How to perform cython files compilation in parallel?


I would like to perform cython files compilation in parallel.

So, I take a look at Cython.Build source file, and find the following signature for cythonize function:

def cythonize(module_list, exclude=None, nthreads=0, aliases=None,
              quiet=False, force=False, language=None,
              exclude_failures=False, **options):

And the following comment about cythonize nthreads option:

"For parallel compilation, set the 'nthreads' option to the number of
concurrent builds."

So I tried to use this option in my setup.py file, like that:

from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils.extension import Extension

EXTENSIONS = [Extension(...)
              ...
              Extension(...)]

setup(name='...',
      ...
      ext_modules=cythonize(EXTENSIONS, nthreads=8),
      ...)

But my .pyx files are still compiled sequentially using 1 thread.

I do not understand what I am doing wrong here and how to use nthreads option to perform cythonize compilation in parallel ?


Solution

  • I finally found a solution to compile my cython file in parallel:

    from setuptools import setup
    from Cython.Build import cythonize
    from Cython.Distutils.extension import Extension
    
    NB_COMPILE_JOBS = 8
    
    EXTENSIONS = [Extension(...)
                  ...
                  Extension(...)]
    
    def setup_given_extensions(extensions):
        setup(name='...',
              ...
              ext_modules=cythonize(extensions),
              ...)
    
    def setup_extensions_in_sequential():
        setup_given_extensions(EXTENSIONS)
    
    def setup_extensions_in_parallel():
        cythonize(EXTENSIONS, nthreads=NB_COMPILE_JOBS)
        pool = multiprocessing.Pool(processes=NB_COMPILE_JOBS)
        pool.map(setup_given_extensions, EXTENSIONS)
        pool.close()
        pool.join()
    
    if "build_ext" in sys.argv:
        setup_extensions_in_parallel()
    else:
        setup_extensions_in_sequential()