kubernetesprometheuscadvisor

Monitor only one namespace pods - Prometheus & Kubernetes & cadvisor


I have deployed istio on kubernetes, and I installed prometheus from istio addons. My goal is to only monitor some pods of one application(such as all pods of bookinfo application). The job definition for monitoring pods is as below:

    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true

My problem is that I don't know how to monitor only one namespace's pods. For example, I deploy the bookinfo application in a namespace named Book. I only want the metrics of pods from namespace Book. However, prometheus will collect all pods metrics of the nodes. Instead of changing annotations of the application like Monitor only one namespace metrics - Prometheus with Kubernetes, I want know if there is a method to select only one namespace by changing the job definition above. Or is there some way to choose the monitor pods by it's labels?


Solution

  • The following will match all target pods with label: some_label with any value.

    relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_pod_label_some_label]
        regex: (.*)
    

    If you want to keep targets with label: monitor and value: true you would do:

    relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_pod_label_monitor]
        regex: true
    

    All pods that don't match it will be dropped from scraping.

    The same you should be able to do for namespaces:

    relabel_configs:
      - action: keep
        source_labels: [__meta_kubernetes_namespace]
        regex: Book
    

    EDIT >

    is there a way to change the [container_label_io_kubernetes_container_name] labels into "container_name"?

    Try this:

    relabel_configs:
      - action: replace
        source_labels: [container_label_io_kubernetes_container_name]
        target_label: container_name
    

    It's all explained in prometheus docs about configuration