pythonrunit

Catching signals sent by runit (sv stop <id>)

I have a python program that is running a number of worker processes. Because this has to be handled properly to avoid orphaned processes, I have implemented a signal handler to shutdown all worker processes.

The program starts more or less like this:

  1. Start process pool (starts X number of workers)
  2. Register signal handlers (signal.signal(signal.SIGTERM, my_signal_handler)). I also add another signal handler for SIGINT with the same handler.
  3. Start seperate thread polling backend (database) and add the tasks to process pool.
  4. On main thread, poll process pool for results (there is a result multiprocessing.Queue that the individual workers add the results to).

The idea is that the two seperate threads started in 3 and 4 keep the tasks running through the machinery.

If I start this manually and call kill -15 <pid> or kill -2 <pid> it correctly shuts everything down, waits for processes to join(). Reading from the documentation, runit sends a TERM to the process, followed by CONT. However, running this under runit, it simply shows the standard ok: down: <my_program>: 1s, normally up, but the process is still running in the background (even the main process, it is UNTOUCHED).

If I then afterwards go out and manually kill the process, I can see in the log file that it shuts down correctly. What am I doing wrong? It seems that runit ONLY kills the 3-line shell script I created to activate the virtualenv, but leaves the actual python process behind.

Even if I run the "run" script directly, I can either run kill or Ctrl+C (same as SIGINT) and it shuts down correctly.

Solution

  • Okay, so after some extensive testing I figured it out.

    Runit will send the kill signal to the run script, which does not propogate it by default. What you need to make sure is that you call exec python yourscript.py in the end. Similarly, if your run script calls another shell script (ie. one that activates your virtualenv or similar), it must do so with exec as well.

    Samples:

    run:

    #!/bin/sh
umask 002
2>&1
exec chpst -uanalytics cliscript router

    cliscript:

    #!/bin/sh

# Resolve script path, assuming that the script resides in $(ABSPATH)/bin
SCRIPTPATH="$0"
if [ -h "$SCRIPTPATH" ]; then
    SCRIPTPATH=$(readlink -e "$0")
fi
ABSPATH=$(dirname "$(cd "$(dirname "$SCRIPTPATH")"; pwd -L)")

# Load the virtual environment
source "$ABSPATH/venv/bin/activate"

# Set up environment
export PYTHONUNBUFFERED=1

exec python "$ABSPATH/bin/processing-cli.py" $@

    Take note of the exec being called when we "pass" control to the next script or python itself.