kubernetesgoogle-kubernetes-enginestackdriverhpa

Horizontal pod autoscaling using a logging custom metric in GKE


I would like to scale my deployment based on a custom logging metric, but I'm not able to make that work, I created already the custom metric and I'm also able to see it in the metric explorer but for some reason the stackdriver adapter is not able to get the metric values.

Custom logging metric

Custom metric in the metric explorer

Custom metric name in metric explorer

This is my hpa.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - external:
      metricName: logging.googleapis.com|user|http_request_custom
      targetValue: "20"
    type: External
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx

But i'm always getting the following error:

"unable to get external metric default/logging.googleapis.com|user|http_request_custom/nil: unable to fetch metrics from external metrics API: the server could not find the requested resource (get logging.googleapis.com|user|http_request_custom.external.metrics.k8s.io)"

Should i do something different?? any idea?


Solution

  • Not sure you have created the service account and granted access to the adapter however there is two models of custom metrics adapter. Legacy adapter and new resource version.

    If adapter is up and running did you check the logs of POD ?

    New resource model to install :

    kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
    

    ref yaml you can use this way further metrics

    apiVersion: autoscaling/v2beta2
    kind: HorizontalPodAutoscaler
    metadata:
      name: pubsub
    spec:
      minReplicas: 1
      maxReplicas: 5
      metrics:
      - external:
          metric:
           name: pubsub.googleapis.com|subscription|num_undelivered_messages
           selector:
             matchLabels:
               resource.labels.subscription_id: echo-read
          target:
            type: AverageValue
            averageValue: 2
        type: External
      scaleTargetRef:
        apiVersion: apps/v1
        kind: Deployment
        name: pubsub
    

    Ref : https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#pubsub_4