kuberneteskubernetes-podtermination

Kubernetes multi-container pod termination process


I have a multi-container pod in k8s, let's call them A and B. When stopping the pod, A must stop before B because A needs B until it's off.

To do that, I registered a preStop hook on A so A can gracefully stop before B.

However I'm not sure this is a good solution, because I miss some information I can't find in k8s documentation:

What happens when a multi-container pod is stopped?

In the second case, preStop is useless for what I want to do as B will be instantly killed.


Solution

  • Typically, during pod deletion, the container runtime sends a TERM signal to the main process in each container.

    According to the official documentation:

    1. If one of the Pod's containers has defined a preStop hook, the kubelet runs that hook inside of the container.

    2. The kubelet triggers the container runtime to send a TERM signal to process 1 inside each container.

    This numeration can confuse - looks like TERM signal will be sent only after preStop hook will be finished. I decided to check the order of work with a simple example below.

    apiVersion: v1
    kind: Pod
    metadata:
      name: lifecycle-demo
    spec:
      restartPolicy: Never
      volumes:
      - name: config
        configMap:
          name: nginx-conf
      containers:
      - name: container-1
        image: nginx
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sleep","15"]
        ports:
        - containerPort: 80
      - name: container-2
        image: nginx
        ports:
        - containerPort: 81
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d
      terminationGracePeriodSeconds: 30
    

    Container-1 has preStop hook for 15 seconds delay. I've connected to both containers to see behavior during pod deletion.

    Result

    After pod deletion:

    1. Container-1 worked for 15 seconds, before the connection was lost

    2. Container-2 immediately lost connection

    Conclusion

    If the container has a preStop hook, it will try to execute it. Only then it will receive TERM signal. The main condition in this case: the grace period has not expired.

    If the container doesn't have a preStop hook, it will receive TERM signal immediately after the command to remove the pod. Thus, it will not wait whilepreStop hook will be executed for another container.

    Note: The containers in the Pod receive the TERM signal at different times and in an arbitrary order. If the order of shutdowns matters, consider using a preStop hook to synchronize.