I'm writing a monitor service in Python that monitors another service and while the monitor & scheduling part works fine, I have a hard time figuring out how to do a proper shutdown of the service using a SIGINT signal send to the Docker container. Specifically, the service should catch the SIGINT from either a docker stop or a Kubernetes stop signal, but so far it doesn't. I have reduced the issue to a minimal test case which is easy to replicate in Docker:
import signal
import sys
import time
class MainApp:
def __init__(self):
self.shutdown = False
signal.signal(signal.SIGINT, self.exit_gracefully)
signal.signal(signal.SIGTERM, self.exit_gracefully)
def exit_gracefully(self, signum, frame):
print('Received:', signum)
self.shutdown = True
def start(self):
print("Start app")
def run(self):
print("Running app")
time.sleep(1)
def stop(self):
print("Stop app")
if __name__ == '__main__':
app = MainApp()
app.start()
# This boolean flag should flip to false when a SIGINT or SIGTERM comes in...
while not app.shutdown:
app.run()
else: # However, this code gets never executed ...
app.stop()
sys.exit(0)
And the corresponding Dockerfile, again minimalistic:
FROM python:3.8-slim-buster
COPY test/TestGS.py .
STOPSIGNAL SIGINT
CMD [ "python", "TestGS.py" ]
I opted for Docker because the Docker stop command is documented to issue a SIGINT signal, waits a bit, and then issues a SIGKILL. This should be an ideal test case.
However, when starting the docker container with an interactive shell attached, and stopping the container from a second shell, the stop() code never gets executed. Verifying the issue, a simple:
$ docker inspect -f '{{.State.ExitCode}}' 64d39c3b
Shows exit code 137 instead of exit code 0.
Apparently, one of two things is happening. Either the SIGTERM signal isn't propagated into the container or Python runtime and this might be true because the exit_gracefully function isn't called apparently otherwise we would see the printout of the signal. I know that you have to be careful about how to start your code from within Docker to actually get a SIGINT, but when adding the stop signal line to the Dockerfile, a global SIGINT should be issued to the container, at least to my humble understanding reading the docs.
Or, the Python code I wrote isn't catching any signal at all. Either way, I simply cannot figure out why the stop code never gets called. I spent a fair amount of time researching the web, but at this point, I feel I'm running circles, Any idea how to solve the issue of correctly ending a python script running inside docker using a SIGINT signal?
Thank you
Marvin
Solution:
The app must run as PID 1 inside docker to receive a SIGINT. To do so, one must use ENTRYPOINT instead of CMD. The fixed Dockerfile:
FROM python:3.8-slim-buster
COPY test/TestGS.py .
ENTRYPOINT ["python", "TestGS.py"]
Build the image:
docker build . -t python-signals
Run the image:
docker run -it --rm --name="python-signals" python-signals
And from a second terminal, stop the container:
docker stop python-signals
Then you get the expected output:
Received SIGTERM signal
Stop app
It seems a bit odd to me that Docker only emits SIGTERMS to PID 1, but thankfully that's relatively easy to fix. The article below was most helpful to solve this issue.
https://itnext.io/containers-terminating-with-grace-d19e0ce34290