kuberneteskubernetes-podreadinessprobe

What happens when kubernetes readiness-probe return false? How long to wait?


What happens when Kubernetes readiness-probe returns false? Does Kubernetes restart that pod after timeout? How long Kubernetes waits for readiness?


Solution

  • Readiness probe doesn't restart the pod/container, readiness probe determines that the container is ready to serve traffic. If the container is probed and considered not "ready", the container will be removed from the endpoints and traffic wont be sent to it, until it is ready again.

    [1] https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-readiness-probes

    [2] https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes

    [3] kubectl explain pod.spec.containers.readinessProbe

    KIND:     Pod
    VERSION:  v1
    
    RESOURCE: readinessProbe <Object>
    
    DESCRIPTION:
         Periodic probe of container service readiness. Container will be removed
         from service endpoints if the probe fails. Cannot be updated. More info:
         https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
    
         Probe describes a health check to be performed against a container to
         determine whether it is alive or ready to receive traffic.
    
    FIELDS:
       exec <Object>
         One and only one of the following should be specified. Exec specifies the
         action to take.
    
       failureThreshold <integer>
         Minimum consecutive failures for the probe to be considered failed after
         having succeeded. Defaults to 3. Minimum value is 1.
    
       httpGet  <Object>
         HTTPGet specifies the http request to perform.
    
       initialDelaySeconds  <integer>
         Number of seconds after the container has started before liveness probes
         are initiated. More info:
         https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
    
       periodSeconds    <integer>
         How often (in seconds) to perform the probe. Default to 10 seconds. Minimum
         value is 1.
    
       successThreshold <integer>
         Minimum consecutive successes for the probe to be considered successful
         after having failed. Defaults to 1. Must be 1 for liveness and startup.
         Minimum value is 1.
    
       tcpSocket    <Object>
         TCPSocket specifies an action involving a TCP port. TCP hooks not yet
         supported
    
       timeoutSeconds   <integer>
         Number of seconds after which the probe times out. Defaults to 1 second.
         Minimum value is 1. More info:
         https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
    

    Let's use the default readiness probe from the documentation:

    cat pod.yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        readinessProbe:
          exec:
            command:
            - cat
            - /tmp/healthy
          initialDelaySeconds: 5
          periodSeconds: 5
    

    To perform a probe, the kubelet executes the command cat /tmp/healthy in the target container. If the command succeeds, it returns 0, then the container is ready and can "serve". if the command returns anything but 0, container is not healthy.

    Since this file doesn't exist in the container from the start, when the pod starts, it is going to be very unhealthy.

    date && k get pods nginx
    Thu  2 Dec 2021 19:08:43 AST
    NAME    READY   STATUS    RESTARTS   AGE
    nginx   0/1     Running   0          66s
    

    now, lets exec into it and create the file, so that the command succeeds.

    k exec -it nginx -- bash
    root@nginx:/# touch /tmp/healthy
    root@nginx:/# exit
    exit
    

    checking again:

    date && k get pods nginx
    Thu  2 Dec 2021 19:09:26 AST
    NAME    READY   STATUS    RESTARTS   AGE
    nginx   1/1     Running   0          110s
    

    removing again:

    k exec -it nginx -- bash
    root@nginx:/# rm /tmp/healthy
    root@nginx:/# exit
    exit
    

    checking:

    date && k get pods nginx
    Thu  2 Dec 2021 19:09:53 AST
    NAME    READY   STATUS    RESTARTS   AGE
    nginx   0/1     Running   0          2m17s