I've got a spring-boot application deployed in kubernetes, with actuator on port 8081
. The main application is working on port 8080
. When I describe the instance, the checks are failing:
$ kubectl describe pod app-844d96f469-8vkbn
...
Warning Unhealthy 29s (x4 over 48s) kubelet Readiness probe failed: Get "http://192.168.13.189:8081/actuator/health/readiness": dial tcp 192.168.13.189:8081: connect: connection refused
Warning Unhealthy 29s (x2 over 39s) kubelet Liveness probe failed: Get "http://192.168.13.189:8081/actuator/health/liveness": dial tcp 192.168.13.189:8081: connect: connection refused
When I ssh into the pod, the checks show that the application is healthy, both on localhost and the ip addresses from the error message:
$ kubectl exec -it pod/app-844d96f469-8vkbn -- /bin/sh
# curl localhost:8081/actuator/health/liveness
{"status":"UP"}
# curl localhost:8081/actuator/health/readiness
{"status":"UP"}
# curl 192.168.13.189:8081/actuator/health/liveness
{"status":"UP"}
# curl 192.168.13.189:8081/actuator/health/readiness
{"status":"UP"}
The application was healthy within the 10*10=100 second window, so this isn't just an issue of adjusting the threshold and period.
This is the kubernetes config for the checks from the deployment:
livenessProbe:
httpGet:
path: "/actuator/health/liveness"
port: 8081
failureThreshold: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: "/actuator/health/readiness"
port: 8081
failureThreshold: 10
periodSeconds: 10
I'd appreciate any help with this issue!
Edit: Adding the Dockerfile, since Thomas asked if 8081 was exposed on the image. I didn't have to do anything for port 8080 to be exposed, but maybe the question wasn't related to the Dockerfile.
FROM eclipse-temurin:17.0.5_8-jre
RUN mkdir /opt/app
ARG JAR_FILE
COPY ${JAR_FILE} /opt/app/app.jar
ENTRYPOINT ["java","-jar","/opt/app/app.jar"]
I suppose that your application takes too long to become ready before it fails due to liveness probe. Spring boot / Java applications do heavy initialization on startup and when CPU is limited, it is slow and takes a long time.
Here is a good explanation: https://faun.pub/java-application-optimization-on-kubernetes-on-the-example-of-a-spring-boot-microservice-cf3737a2219c
A simple service takes 80 seconds on 0.5 CPU while it only takes 12 seconds on 3 CPUs.
If you hava CPU limit set, just leave it out. Only set a request.
And maybe also configure an initialDelaySeconds: 60
or so for the liveness probe.