I've been trying to drop unused metrics (before scrape) to lighten the load on a Prometheus cluster via relabel_configs:
- job_name: nginx-ingress-controller-metrics
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- <some-namespace>
relabel_configs:
# single
- source_labels: [__name__]
regex: nginx_ingress_controller_response_size_bucket
action: drop
# multiple
- source_labels: [__name__]
regex: nginx_ingress_controller_(request_size_bucket|response_duration_seconds_bucket|bytes_sent_bucket|bytes_sent_sum)
action: drop
However, I am not seeing any of these metrics dropped (before scrape) and they are scraped and ingested as I can query them via PromQL.
I've tried many variations of the same config (i.e. quotes, parentheses, etc.). Any idea why these targets are still scraped/ingested?
What you declare in relabel_configs happens before Prometheus even makes a connection to an exporter. No connection means there is no metrics and no labels yet, so you can't drop metrics by __name__ at this point.
What you actually need to use is metric_relabel_configs. This relabeling step happens after Prometheus has retrieved data from an exporter, but before data is actually saved into the database. Here's an example:
metric_relabel_configs:
- source_labels: [__name__]
regex: my_useless_metric.*
action: drop
If you're curious "what's the point of relabel_configs then?", relabel_configs operate metrics on the scale of hosts. Using service discovery information (which is available at this point) you can use relabeling to change address, port or to make a decision whether this instance should be scraped at all.