I have a multi-container pod in k8s, let's call them A and B. When stopping the pod, A must stop before B because A needs B until it's off.
To do that, I registered a preStop
hook on A so A can gracefully stop before B.
However I'm not sure this is a good solution, because I miss some information I can't find in k8s documentation:
What happens when a multi-container pod is stopped?
preStop
hooks are called, then when they are all over all containers receive SIGTERM
, orpreStop
if they have one or directly SIGTERM
if they don't?In the second case, preStop
is useless for what I want to do as B will be instantly killed.
Typically, during pod deletion, the container runtime sends a TERM signal to the main process in each container.
According to the official documentation:
If one of the Pod's containers has defined a
preStop
hook, the kubelet runs that hook inside of the container.The kubelet triggers the container runtime to send a TERM signal to process 1 inside each container.
This numeration can confuse - looks like TERM signal will be sent only after preStop
hook will be finished.
I decided to check the order of work with a simple example below.
apiVersion: v1
kind: Pod
metadata:
name: lifecycle-demo
spec:
restartPolicy: Never
volumes:
- name: config
configMap:
name: nginx-conf
containers:
- name: container-1
image: nginx
lifecycle:
preStop:
exec:
command: ["/bin/sleep","15"]
ports:
- containerPort: 80
- name: container-2
image: nginx
ports:
- containerPort: 81
volumeMounts:
- name: config
mountPath: /etc/nginx/conf.d
terminationGracePeriodSeconds: 30
Container-1 has preStop
hook for 15 seconds delay.
I've connected to both containers to see behavior during pod deletion.
Result
After pod deletion:
Container-1 worked for 15 seconds, before the connection was lost
Container-2 immediately lost connection
Conclusion
If the container has a preStop
hook, it will try to execute it. Only then it will receive TERM signal. The main condition in this case: the grace period has not expired.
If the container doesn't have a preStop
hook, it will receive TERM signal immediately after the command to remove the pod. Thus, it will not wait whilepreStop
hook will be executed for another container.
Note: The containers in the Pod receive the TERM signal at different times and in an arbitrary order. If the order of shutdowns matters, consider using a
preStop
hook to synchronize.