pythonmultiprocessingpython-multiprocessing

Python multiprocessing Pool not working KeyboardInterrupt


When running the following simple multiprocessing.Pool code, the KeyboardInterrupt does not stop the python program.

Code

import multiprocessing as mp
import time

print(f"INIT {__name__}")


def worker(a):
    time.sleep(999)


def main():
    with mp.Pool(2) as p:
        for _ in p.imap(worker, range(3)):
            ...


if __name__ == "__main__":
    main()

Console Output

INIT __main__
INIT __mp_main__
INIT __mp_main__
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-1:
Traceback (most recent call last):
Traceback (most recent call last):
  File "Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "C:\WIN_PROJEKTE\informatik\synthetic-data-generation\src\test2.py", line 8, in worker
    time.sleep(999)
KeyboardInterrupt
  File "Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "test2.py", line 8, in worker
    time.sleep(999)
KeyboardInterrupt
INIT __mp_main__
INIT __mp_main__

As you can see, when I hit CTRL+C, the two workers receive it and stop. But the main process appaerently does not receive it and the two multiprocessing threads are spawned again.

What I expect

When I press CTRL+C, the main process should stop.
Instead, the multiprocessing workers are stopped and spawned again.

When I hit CTRL+C a second time, the same problem occurs

I tried


Solution

  • I found that catching the KeyboardInterrupt in both the workers and main works correctly, but leave out either try/except and main continues to restart the workers.

    import multiprocessing as mp
    import time
    
    print(f'INIT {__name__}')
    
    def worker(a):
        try:
            time.sleep(999)
        except KeyboardInterrupt:
            print('worker ctrl-c')
    
    def main():
        try:
            with mp.Pool(2) as p:
                for _ in p.imap(worker, range(3)):
                    pass
        except KeyboardInterrupt:
            print('main ctrl-c')
    
    if __name__ == '__main__':
        main()
    

    Output (hitting ctrl-c after the INITs):

    INIT __main__
    INIT __mp_main__
    INIT __mp_main__
    worker ctrl-c
    worker ctrl-c
    main ctrl-c