I have deployed istio on kubernetes, and I installed prometheus from istio addons. My goal is to only monitor some pods of one application(such as all pods of bookinfo application). The job definition for monitoring pods is as below:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: kubernetes-nodes-cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- replacement: kubernetes.default.svc:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
My problem is that I don't know how to monitor only one namespace's pods. For example, I deploy the bookinfo application in a namespace named Book. I only want the metrics of pods from namespace Book. However, prometheus will collect all pods metrics of the nodes. Instead of changing annotations of the application like Monitor only one namespace metrics - Prometheus with Kubernetes, I want know if there is a method to select only one namespace by changing the job definition above. Or is there some way to choose the monitor pods by it's labels?
The following will match all target pods with label: some_label
with any value.
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_pod_label_some_label]
regex: (.*)
If you want to keep targets with label: monitor
and value: true
you would do:
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_pod_label_monitor]
regex: true
All pods that don't match it will be dropped from scraping.
The same you should be able to do for namespaces:
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_namespace]
regex: Book
EDIT >
is there a way to change the [container_label_io_kubernetes_container_name] labels into "container_name"?
Try this:
relabel_configs:
- action: replace
source_labels: [container_label_io_kubernetes_container_name]
target_label: container_name
It's all explained in prometheus docs about configuration