I have this config in my vmagent:
global:
scrape_interval: 60s
scrape_timeout: 60s
external_labels:
server_name: vmagent
scrape_configs:
- job_name: "kafka_exporter"
file_sd_configs:
- files:
- kafka_exporter.yml
metric_relabel_configs:
- if: '{__name__="kafka_consumergroup_lag_sum"}'
target_label: foo
replacement: 3
Im trying to add label to only one metric. There it is:
kafka_consumergroup_lag_sum{consumergroup="test",topic="elk"} 0
But if I search metrics im my VictoriaMetrics (remote write to vmagent), its no-one metrics with this label. Here they are:
sum by(__name__)({__name__=~".+",foo="3"}):
kafka_consumergroup_lag_sum{}
kafka_consumergroup_members{}
kafka_exporter_build_info{}
kafka_topic_partition_current_offset{}
kafka_topic_partition_in_sync_replica{}
kafka_topic_partition_leader{}
kafka_topic_partition_leader_is_preferred{}
kafka_topic_partition_oldest_offset{}
kafka_topic_partition_replicas{}
kafka_topic_partition_under_replicated_partition{}
kafka_topic_partitions{}
process_cpu_seconds_total{}
process_max_fds{}
process_open_fds{}
process_resident_memory_bytes{}
process_start_time_seconds{}
process_virtual_memory_bytes{}
process_virtual_memory_max_bytes{}
promhttp_metric_handler_requests_in_flight{}
promhttp_metric_handler_requests_total{}
What am I doing wrong? Why other metrics have same label?
If I try to do the same for another metric (kafka_topic_partitions), there is no such problem (Its not true! See "p.s."). Config fully same:
- if: '{__name__="kafka_topic_partitions"}'
target_label: foo
replacement: 3
p.s. I found the pattern. Tags are added to all metric after the selected (in example: kafka_consumergroup_lag_sum). If I select kafka_topic_partitions, then list is:
sum by(__name__)({__name__=~".+",foo="3"}):
kafka_topic_partitions{}
process_cpu_seconds_total{}
process_max_fds{}
process_open_fds{}
process_resident_memory_bytes{}
process_start_time_seconds{}
process_virtual_memory_bytes{}
process_virtual_memory_max_bytes{}
promhttp_metric_handler_requests_in_flight{}
promhttp_metric_handler_requests_total{}
It looks as if works once to determine the cutting point.
How I can add label to only one metric?
I opend issue on GitHub about similar problem (I wanted to filter metrics, tags was a tool): https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6080
This is not a bug, it really works like that, it’s just that it is not written anywhere. Not all contributors even know about it, but https://github.com/f41gh7 is a real expert of VM!
In my case its need to use to filter metrics.
metric_relabel_configs:
- source_labels: [__name__]
action: keep
regex: '^kafka_consumergroup_lag_sum$'
This does not answer how to add tags, but for me the question is resolved. It is not difficult to figure it out with tags, knowing all this.