prometheusprometheus-alertmanager

Drop label in Prometheus, but keep it for alertmanager

Is there a way to configure Prometheus to drop (don't save to database) certain label, but keep and pass it to alertmanager when alert arises?

This label is only matters in the moment when alert is active. I don't want to store it's history.

E.g. my_instance_id should be kept for alertmanager to use it in alerts text, but should not be saved into the Prometheus database:

up{instance="myhostname", job="myjob", my_instance_id="12345"}

I know it is possible to add/modify labels for alertmanager only with alert_relabel_configs, but if I drop certain label in Prometheus (to avoid saving it in database) with metric_relabel_configs - it would be no such label for alert_relabel_configs to work with.

Solution

  • No, this is not possible.

    Alert rules are evaluated on the stored data, not freshly scrapped ones. And thus can only use labels that are present in storage.