pythongnu-parallelatexit

GNU Parallel and Python atexit

I was trying to run a python script with GNU parallel. Everything seems to work, except for the atexit routine used inside the python script. It seems, after ctrl+c, parallel is killing the python process without giving python a chance to call the registered atexit routine. How to make parallel a bit nicer towards the child processes?

Here is an example to show the behaviour.

test_signal.py:

#!/usr/bin/env python3

import time
import sys
import atexit

def cleanup():
    print('cleanup called', flush=True)

atexit.register(cleanup)

time.sleep(60)
print('completed process', sys.argv[1])

Tested with the command:

chmod +x test_signal.py
./test_signal.py 1 # this works as expected when using ^C
parallel -j 4 ./test_signal.py {} ::: $(seq 1 12) # this one does not
Solution

  • GNU Parallel kills jobs as specified in --termseq. It defaults to:

    TERM,200,TERM,100,TERM,50,KILL,25

    So SIGTERM, wait 200 ms, SIGTERM, wait 100 ms, SIGTERM, wait 50 ms, SIGKILL, wait 25 ms. If the process dies while waiting, GNU Parallel ignores the rest of the sequence - no need to kill a dead horse.

    Change --termseq to the signals you want sent and the time between them.

    CTRL-C sends SIGINT, so this should work:

    # Give 1 second to clean up, then kill
parallel --termseq INT,1000,KILL,25 ...

    Just be aware that you will not see the output unless you use --ungroup.