prometheusmonitoringpromqlvictoriametricsmetricsql

VictoriaMetrics (VM Agent) relable metrics pattern


I have this config in my vmagent:

global:
  scrape_interval: 60s
  scrape_timeout: 60s
  external_labels:
    server_name: vmagent

scrape_configs:
  - job_name: "kafka_exporter"
    file_sd_configs:
    - files:
      - kafka_exporter.yml
    metric_relabel_configs:
      - if: '{__name__="kafka_consumergroup_lag_sum"}'
        target_label: foo
        replacement: 3

Im trying to add label to only one metric. There it is:

kafka_consumergroup_lag_sum{consumergroup="test",topic="elk"} 0

But if I search metrics im my VictoriaMetrics (remote write to vmagent), its no-one metrics with this label. Here they are:

sum by(__name__)({__name__=~".+",foo="3"}):

kafka_consumergroup_lag_sum{}
kafka_consumergroup_members{}
kafka_exporter_build_info{}
kafka_topic_partition_current_offset{}
kafka_topic_partition_in_sync_replica{}
kafka_topic_partition_leader{}
kafka_topic_partition_leader_is_preferred{}
kafka_topic_partition_oldest_offset{}
kafka_topic_partition_replicas{}
kafka_topic_partition_under_replicated_partition{}
kafka_topic_partitions{}
process_cpu_seconds_total{}
process_max_fds{}
process_open_fds{}
process_resident_memory_bytes{}
process_start_time_seconds{}
process_virtual_memory_bytes{}
process_virtual_memory_max_bytes{}
promhttp_metric_handler_requests_in_flight{}
promhttp_metric_handler_requests_total{}

What am I doing wrong? Why other metrics have same label?

If I try to do the same for another metric (kafka_topic_partitions), there is no such problem (Its not true! See "p.s."). Config fully same:

      - if: '{__name__="kafka_topic_partitions"}'
        target_label: foo
        replacement: 3

p.s. I found the pattern. Tags are added to all metric after the selected (in example: kafka_consumergroup_lag_sum). If I select kafka_topic_partitions, then list is:

sum by(__name__)({__name__=~".+",foo="3"}):

kafka_topic_partitions{}
process_cpu_seconds_total{}
process_max_fds{}
process_open_fds{}
process_resident_memory_bytes{}
process_start_time_seconds{}
process_virtual_memory_bytes{}
process_virtual_memory_max_bytes{}
promhttp_metric_handler_requests_in_flight{}
promhttp_metric_handler_requests_total{}

It looks as if works once to determine the cutting point.

How I can add label to only one metric?


Solution

  • I opend issue on GitHub about similar problem (I wanted to filter metrics, tags was a tool): https://github.com/VictoriaMetrics/VictoriaMetrics/issues/6080

    This is not a bug, it really works like that, it’s just that it is not written anywhere. Not all contributors even know about it, but https://github.com/f41gh7 is a real expert of VM!

    In my case its need to use to filter metrics.

        metric_relabel_configs:
        - source_labels: [__name__]
          action: keep
          regex: '^kafka_consumergroup_lag_sum$'
    

    This does not answer how to add tags, but for me the question is resolved. It is not difficult to figure it out with tags, knowing all this.