[SOLVED] Kubernetes - some HPA's are scaling up based on other HPA's metrics not their own

Kubernetes - some HPA's are scaling up based on other HPA's metrics not their own

I wonder if anyone is able to help me understand what im doing wrong here.. I have 3 deployments in my namespace, each of them has a Horizontal Pod Autoscaler configured. However, despite each of the HPA's being configured with a target of its respective deployment, They all seem to be reacting to the same CPU metric

$ kubectl get hpa -n my-namespace
NAME       REFERENCE         TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
app1-hpa   Deployment/app1   54%/100%   2         5         5          11h
app2-hpa   Deployment/app2   54%/100%   10        40        39         11h
app3-hpa   Deployment/app3   54%/100%   10        50        39         11h

In my example, app3 is the only one that is busy, but if you look at the TARGETS column, the % utilisation for all 3 HPA's is being calculated the same, so they have ALL scaled up... app1 for example, which is completely idle (using only 1m of cpu per pod) has scaled up to 5 because the metric says that its on 54%/100% ....

What im trying to achieve is that each HPA reacts ONLY to the CPU metrics of the deployment its paired with. therefore in the above example, app1 would clearly stay at 2 instances

my HPA configuration looks like this (below is an example for app1)

---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: app1-hpa
  namespace: my-namespace
spec:
  maxReplicas: 5
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app1
  targetCPUUtilizationPercentage: 100

Here is the deployment code for app1 (app2 and app3 are the same all but in name)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app1
  labels:
    app: my-namespace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: app
    spec:
      imagePullSecrets:
      - name: container-registry-access
      containers:
      - name: app
        image: "path_to_registry/location/imagename"
        resources:
          requests:
            memory: 512Mi
            cpu: 500m
          limits:
            memory: 512Mi
            cpu: 1000m
        env:
        - name: LOG_LEVEL
          value: info
        command: ["/entrypoint-app1.sh"]
        imagePullPolicy: Always
      restartPolicy: Always

Does anyone know what im doing wrong here? it seems to be scaling on an overall CPU average of all pods or something like that? If app3 is really busy , i dont want app1 and app2 to scale up as well when they are actually idle

Any help on this would be greatly appreciated

Solution

if all of your deployments are the same except for the name i would suggest that you change the spec.template.metadata.labels.app and the spec.selector.matchLabels.app to correspond to the right application, meaning that each of those value for app1 would be app1 and not app. my guess that deployments think that all the apps are the same and this is why the cpu average the same for all.