spring-bootkubernetesspring-boot-actuatorkubernetes-health-check

Kubernetes liveness/readiness checks with spring boot application on separate port


I've got a spring-boot application deployed in kubernetes, with actuator on port 8081. The main application is working on port 8080. When I describe the instance, the checks are failing:

$ kubectl describe pod app-844d96f469-8vkbn
...
  Warning  Unhealthy  29s (x4 over 48s)  kubelet            Readiness probe failed: Get "http://192.168.13.189:8081/actuator/health/readiness": dial tcp 192.168.13.189:8081: connect: connection refused
  Warning  Unhealthy  29s (x2 over 39s)  kubelet            Liveness probe failed: Get "http://192.168.13.189:8081/actuator/health/liveness": dial tcp 192.168.13.189:8081: connect: connection refused

When I ssh into the pod, the checks show that the application is healthy, both on localhost and the ip addresses from the error message:

$ kubectl exec -it pod/app-844d96f469-8vkbn -- /bin/sh
# curl localhost:8081/actuator/health/liveness
{"status":"UP"}
# curl localhost:8081/actuator/health/readiness
{"status":"UP"}
# curl 192.168.13.189:8081/actuator/health/liveness
{"status":"UP"}
# curl 192.168.13.189:8081/actuator/health/readiness
{"status":"UP"}

The application was healthy within the 10*10=100 second window, so this isn't just an issue of adjusting the threshold and period.

This is the kubernetes config for the checks from the deployment:

        livenessProbe:
          httpGet:
            path: "/actuator/health/liveness"
            port: 8081
          failureThreshold: 10
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: "/actuator/health/readiness"
            port: 8081
          failureThreshold: 10
          periodSeconds: 10

I'd appreciate any help with this issue!


Edit: Adding the Dockerfile, since Thomas asked if 8081 was exposed on the image. I didn't have to do anything for port 8080 to be exposed, but maybe the question wasn't related to the Dockerfile.

FROM eclipse-temurin:17.0.5_8-jre
RUN mkdir /opt/app
ARG JAR_FILE
COPY ${JAR_FILE} /opt/app/app.jar
ENTRYPOINT ["java","-jar","/opt/app/app.jar"]

Solution

  • I suppose that your application takes too long to become ready before it fails due to liveness probe. Spring boot / Java applications do heavy initialization on startup and when CPU is limited, it is slow and takes a long time.

    Here is a good explanation: https://faun.pub/java-application-optimization-on-kubernetes-on-the-example-of-a-spring-boot-microservice-cf3737a2219c

    A simple service takes 80 seconds on 0.5 CPU while it only takes 12 seconds on 3 CPUs.

    If you hava CPU limit set, just leave it out. Only set a request. And maybe also configure an initialDelaySeconds: 60 or so for the liveness probe.