I am setting up a Prometheus alert that will alert on several metrics. It will look something like this:
metric_x > threshold or metric_y > threshold or metric_x > threshold
But if this alert goes off, I would like to include something in the description about which metric got alerted on. Is there a way to know which one of those 3 conditions cause the alarm to go off? Just as a way to add more details in the alarm description? Is the only way to do that is to have separate alarms or include the metric value in the description?
A metric name under the hood is just another label, which means you can insert it into an annotation. Here's how:
- record: foo
expr: 1
- record: bar
expr: 2
- alert: test
expr: foo == 1 or bar == 2
# You probably need just one of these
labels:
name: '{{ .Labels.__name__ }}'
annotations:
name: '{{ .Labels.__name__ }}'