I am reading through the HPA walkthrough available on the kubernetes documentation here. I am unable to get the HPA to scale the deployment when using the AverageValue instead of Utilization.
I am using a 1.25 minikube cluster and have metrics server deployment and patched.
kubectl patch deployment metrics-server -n kube-system --type 'json' -p '[{"op": "add", "path": "/spec/template/spec/containers/0/args/-", "value": "--kubelet-insecure-tls"}]'
I am able to successfully execute kubectl top pods/nodes commands successfully which to me means that the metrics server is working fine.
Gauravs-MBP:K8s alieninvader$ kubectl top pods
NAME CPU(cores) MEMORY(bytes)
load-generator 6m 0Mi
php-apache-5b56f9df94-r74qk 501m 11Mi
Gauravs-MBP:K8s alieninvader$ kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
minikube 719m 11% 834Mi 10%
When I use the Utilization instead of AverageValue, the HPA is able to scale the deployment. I have used the same php-apache deployment and used this as my manifest to create the HPA.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
namespace: default
spec:
maxReplicas: 10
metrics:
- resource:
name: cpu
target:
averageValue: 50
type: AverageValue
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
When i look at the events for the HPA, i see that HPA still calculates this to be within recommended range.
kubectl describe hpa php-apache
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Tue, 20 Dec 2022 13:43:19 -0500
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods: 501m / 50
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
Gauravs-MBP:K8s alieninvader$ kubectl get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 1m/50 1 10 1 37s
php-apache Deployment/php-apache 189m/50 1 10 1 45s
php-apache Deployment/php-apache 501m/50 1 10 1 60s
php-apache Deployment/php-apache 502m/50 1 10 1 90s
php-apache Deployment/php-apache 500m/50 1 10 1 105s
php-apache Deployment/php-apache 501m/50 1 10 1 2m
php-apache Deployment/php-apache 500m/50 1 10 1 2m15s
php-apache Deployment/php-apache 501m/50 1 10 1 2m30s
php-apache Deployment/php-apache 499m/50 1 10 1 2m45s
php-apache Deployment/php-apache 502m/50 1 10 1 3m
php-apache Deployment/php-apache 501m/50 1 10 1 3m15s
php-apache Deployment/php-apache 500m/50 1 10 1 3m30s
php-apache Deployment/php-apache 501m/50 1 10 1 3m45s
php-apache Deployment/php-apache 500m/50 1 10 1 4m
php-apache Deployment/php-apache 502m/50 1 10 1 4m15s
php-apache Deployment/php-apache 500m/50 1 10 1 4m30s
php-apache Deployment/php-apache 501m/50 1 10 1 4m45s
php-apache Deployment/php-apache 500m/50 1 10 1 5m
php-apache Deployment/php-apache 501m/50 1 10 1 5m15s
php-apache Deployment/php-apache 500m/50 1 10 1 5m30s
php-apache Deployment/php-apache 501m/50 1 10 1 5m45s
php-apache Deployment/php-apache 500m/50 1 10 1 6m15s
php-apache Deployment/php-apache 502m/50 1 10 1 6m30s
php-apache Deployment/php-apache 500m/50 1 10 1 6m45s
So why when using AverageValue the HPA considers this to be in range, when clearly the CPU percentage indicated in the kubectl top o/p is way higher. What am i missing here?
In contrast to averageUtilization, averageValue should be a literal value, not a percentage (see MetricTarget). In your setup it's set to 50 cores, which is clearly much higher then the load on the pod. Did you mean 500m?