prometheusprometheus-alertmanageralerts

Get metric name in Prometheus Alert


I am setting up a Prometheus alert that will alert on several metrics. It will look something like this:

metric_x > threshold or metric_y > threshold or metric_x > threshold

But if this alert goes off, I would like to include something in the description about which metric got alerted on. Is there a way to know which one of those 3 conditions cause the alarm to go off? Just as a way to add more details in the alarm description? Is the only way to do that is to have separate alarms or include the metric value in the description?


Solution

  • A metric name under the hood is just another label, which means you can insert it into an annotation. Here's how:

    - record: foo
      expr: 1
    - record: bar
      expr: 2
    - alert: test
      expr: foo == 1 or bar == 2
      # You probably need just one of these
      labels:
        name: '{{ .Labels.__name__ }}'
      annotations:
        name: '{{ .Labels.__name__ }}'