How to add prefix to metrics using relable_config in prometheus

finally running Harvest2 in a Docker environment with Prometheus and Grafana.

Problem is, there´s also other systems reporting in that same Prometheus DB and NetApp Harvest does not add a prefix on it´s metric names like netapp_ to every netapp metric. It is such a pain in the ass finding the correct metrics this way.

I would like to use the relable_config option of Prometheus_config as a workaround. At the moment i have the following configuration for the harvest pollers:

 - job_name: harvest
    scrape_interval: 1m
    scrape_timeout: 1m
    metrics_path: /metrics
    relabel_configs:
    - action: replace
      source_labels: [__name__]
      regex: (.*)
      target_label: __name__
      replacement: 'netapp_$1'

    - action: keep
      source_labels:
      - "custom_labels"
      - "custom_labels"
      - "custom_labels"
      - "custom_labels"
      regex: '.+;.+;.+;.+'

    file_sd_configs:
     - refresh_interval: 10s
       files:
       - targets/harvest.yml

But this leads to the harvest pollers not showing up in Grafana/Prometheus at all. Any idea on how to add the required prefix?

Solution

The are two problems with this configuration but they have the same cause. Furthermore, it's likely that adding prefix might not be the best idea in this case. Make sure you read the note at the end of this answer.

The problem is that relabel_configs contain relabel configurations that should be applied before scraping. For example, you can change __address__, so that Prometheus will contact some another host instead of that given by service discovery. Naturally, __name__ is not available at this time because Prometheus has not scraped anything yet.

The solution is to move __name__ to metric_relabel_configs. At this step metrics are already collected but not yet ingested and you can change their __name__. Here is an example that adds super_ prefix to all scraped metrics:

metric_relabel_configs:
- source_labels: [__name__]
  target_label: __name__
  replacement: super_$1

Pretty much the same is with the second relabeling in the question:

    - action: keep
      source_labels:
      - "custom_labels"
      - "custom_labels"
      - "custom_labels"
      - "custom_labels"
      regex: '.+;.+;.+;.+'

Metrics are not yet scraped and so there are no custom labels yet. This effectively drops all targets because none has the mentioned labels. If you think you need this relabeling (it is not required for adding a prefix to label names), you have to put it under metric_relabel_configs.

Why adding a prefix to label names might be not the best idea

The problem is that because of different label names you will not be able to use the same dashboard/alerts with renamed metrics. You will have to make separate dashboards for these metrics or use weird and ineffective queries, such as this:

{__name__=~".*my_metric_without_prefix"}

It might be better to simply add a label that will differentiate one set of metrics from the other one. You can either add your own new label or use different jobs to scrape those metrics. In the last case you can distinguish one set of the metrics from the other by job label.