kuberneteskubernetes-apiserver

How to Enable KubeAPI server for HPA Autoscaling Metrics


I am using Kube version v1.13.0. Since Heapster is depreciated from v1.11 I am stuck in enabling the API server for cluster Metrics to implement HPA.

root:~# kubectl get pods,deployment,hpa
NAME                                   READY   STATUS    RESTARTS   AGE
pod/ctauthorization-754c686bc6-6adkx   1/1     Running   0          72s
pod/ctauthorization-754c686bc6-dpsbm   1/1     Running   0          57s
pod/ctauthorization-754c686bc6-qsavs   1/1     Running   0          72s

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/ctauthorization   3/3     3            3           72s

NAME                                                  REFERENCE                    TARGETS                           MINPODS   MAXEODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/ctauthorization   Deployment/ctauthorization   <unknown>/1000Mi, <unknown>/50%   3         5         3          72s

Can someone guide me for step by step enable for the API Metrics server or any Demo video. It would be really helpful to proceed further.

Please let me know if any further information needed.

Thanks Deena


Solution

  • I am able to implement HPA using metrics-server as heapster is depreciated. I have followed the following steps:

    1. Clone the metrics-server github repo: git clone https://github.com/kubernetes-incubator/metrics-server.git

    Go into directory cd deploy/1.8+ and run following yaml files:

    [root@ip-10-0-1-91 1.8+]# kubectl apply -f aggregated-metrics-reader.yaml 
    clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
    [root@ip-10-0-1-91 1.8+]# kubectl apply -f auth-reader.yaml 
    rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
    [root@ip-10-0-1-91 1.8+]# kubectl apply -f auth-delegator.yaml 
    clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
    [root@ip-10-0-1-91 1.8+]# kubectl apply -f metrics-apiservice.yaml 
    apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
    [root@ip-10-0-1-91 1.8+]# kubectl apply -f resource-reader.yaml 
    clusterrole.rbac.authorization.k8s.io/system:metrics-server created
    clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
    [root@ip-10-0-1-91 1.8+]# kubectl apply -f metrics-server-deployment.yaml 
    serviceaccount/metrics-server created
    deployment.extensions/metrics-server created
    [root@ip-10-0-1-91 1.8+]# kubectl apply -f metrics-server-service.yaml 
    service/metrics-server created
    

    Now create a pod you want to test for autoscaling (taken from kubernetes official docs):

    [root@ip-10-0-1-91 auto]#  kubectl run --generator=run-pod/v1 php-apache -- 
    image=k8s.gcr.io/hpa-example --requests=cpu=200m --expose --port=80
    service/php-apache created
    deployment.apps/php-apache created
    

    Now create a autoscale deployment:

    [root@ip-10-0-1-91 auto]# kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
    horizontalpodautoscaler.autoscaling/php-apache autoscaled
    

    Now check the HPA, your metrics are coming or not:

    [root@ip-10-0-1-91 manifests]# kubectl get hpa
    NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
    php-apache   Deployment/php-apache   0%/50%    1         10        1          2m
    

    Now generate load from another window using:

    kubectl run -i --tty load-generator --image=busybox /bin/sh
    

    It will open a sh terminal and you can run a load from that sh terminal using:

    while true; do wget -q -O- http://php-apache.default.svc.cluster.local; done
    

    It will take a minute or so to take enough load on your pod and you see a boom:

    [root@ip-10-0-1-91 manifests]# kubectl get hpa
    NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
    php-apache   Deployment/php-apache   120%/50%   1         10        4          7m
    

    And pods scaling :

    enter image description here

    Hope this helps to get your HPA working.

    EDIT:

    Replace the metrics-server-deployment.yaml file in deploy/1.8+ with the following yaml file:

     apiVersion: v1
     kind: ServiceAccount
     metadata:
       name: metrics-server
       namespace: kube-system
     ---
     apiVersion: extensions/v1beta1
     kind: Deployment
     metadata:
       name: metrics-server
       namespace: kube-system
       labels:
         k8s-app: metrics-server
     spec:
       selector:
         matchLabels:
           k8s-app: metrics-server
       template:
         metadata:
           name: metrics-server
           labels:
             k8s-app: metrics-server
         spec:
           serviceAccountName: metrics-server
           volumes:
           # mount in tmp so we can safely use from-scratch images and/or read-only containers
           - name: tmp-dir
             emptyDir: {}
           containers:
           - command:
             - /metrics-server
             - --metric-resolution=30s
             - --kubelet-insecure-tls
             - --kubelet-preferred-address-types=InternalIP
             name: metrics-server
             image: k8s.gcr.io/metrics-server-amd64:v0.3.1
             imagePullPolicy: Always
             volumeMounts:
             - name: tmp-dir
               mountPath: /tmp
    

    Also, enable the --authentication-token-webhook in kubelet.conf, then you will be able to get the HPA.

    EDIT2: You need to set following properties in the deployment file (in your case it is tomcat) for which you are creating HPA, then only your HPA can fetch metrics from your deployment.

    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"