kuberneteshpa

How to avoid cpu spike on k8s hpa when spring application is loading

I manage a spring applications on K8s.

Pods take about 2~3 minutes for the application to run, and the probe waiting time is set at 4 minutes. enter image description here

enter image description here

After re-deploy an application, the CPU Spike symptom occurs while the Pod is being initialized. So, when Rolling Update is performed, all Pods are scaled up due to the CPU spike symptom. enter image description here

enter image description here

I don't wanna collect a cpu while initialize application. Is there any way to delay collect cpu info?

K8s Version : 1.26.3

Deployment Info

...
livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /probex
            port: 9006
            scheme: HTTP
          initialDelaySeconds: 240
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 1
        name: sample-container
        ports:
        - containerPort: 9006
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /probex
            port: 9006
            scheme: HTTP
          initialDelaySeconds: 240
          periodSeconds: 60
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 8Gi
          requests:
            cpu: 500m
            memory: 8Gi
...

Hpa Info

...
spec:
  maxReplicas: 5
  metrics:
  - resource:
      name: memory
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  - resource:
      name: cpu
      target:
        averageUtilization: 70
        type: Utilization
    type: Resource
  minReplicas: 1
...
Solution

  • You can read about stabilizationWindowSeconds and periodSeconds paramters in the hpa configuration and tune them as per your use case.