pythonmultiprocessingpool

Python Process Pool non-daemonic?


Would it be possible to create a python Pool that is non-daemonic? I want a pool to be able to call a function that has another pool inside.

I want this because deamon processes cannot create process. Specifically, it will cause the error:

AssertionError: daemonic processes are not allowed to have children

For example, consider the scenario where function_a has a pool which runs function_b which has a pool which runs function_c. This function chain will fail, because function_b is being run in a daemon process, and daemon processes cannot create processes.


Solution

  • The multiprocessing.pool.Pool class creates the worker processes in its __init__ method, makes them daemonic and starts them, and it is not possible to re-set their daemon attribute to False before they are started (and afterwards it's not allowed anymore). But you can create your own sub-class of multiprocesing.pool.Pool (multiprocessing.Pool is just a wrapper function) and substitute your own multiprocessing.Process sub-class, which is always non-daemonic, to be used for the worker processes.

    Here's a full example of how to do this. The important parts are the two classes NoDaemonProcess and MyPool at the top and to call pool.close() and pool.join() on your MyPool instance at the end.

    #!/usr/bin/env python
    # -*- coding: UTF-8 -*-
    
    import multiprocessing
    # We must import this explicitly, it is not imported by the top-level
    # multiprocessing module.
    import multiprocessing.pool
    import time
    
    from random import randint
    
    
    class NoDaemonProcess(multiprocessing.Process):
        # make 'daemon' attribute always return False
        def _get_daemon(self):
            return False
        def _set_daemon(self, value):
            pass
        daemon = property(_get_daemon, _set_daemon)
    
    # We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
    # because the latter is only a wrapper function, not a proper class.
    class MyPool(multiprocessing.pool.Pool):
        Process = NoDaemonProcess
    
    def sleepawhile(t):
        print("Sleeping %i seconds..." % t)
        time.sleep(t)
        return t
    
    def work(num_procs):
        print("Creating %i (daemon) workers and jobs in child." % num_procs)
        pool = multiprocessing.Pool(num_procs)
    
        result = pool.map(sleepawhile,
            [randint(1, 5) for x in range(num_procs)])
    
        # The following is not really needed, since the (daemon) workers of the
        # child's pool are killed when the child is terminated, but it's good
        # practice to cleanup after ourselves anyway.
        pool.close()
        pool.join()
        return result
    
    def test():
        print("Creating 5 (non-daemon) workers and jobs in main process.")
        pool = MyPool(5)
    
        result = pool.map(work, [randint(1, 5) for x in range(5)])
    
        pool.close()
        pool.join()
        print(result)
    
    if __name__ == '__main__':
        test()