prometheusalertprometheus-alertmanagerprometheus-operator

value of one alert can be prnted in annotation description of another alert in Alert manager


I have one alert "sc_pcd_g2version" listed in AlertManager. It returns value 25.

I am looking for a way to pull this value "25" and print it into the annotation description of another alert "sc_pcd_encryption_server" in Alert manager.

See in the alert sc_pcd_encryption_server you can see in Bold letter that i am looking for to be included. To use Current alert value we use $labels.value likewise how to pull value of another alert and print it in annotation description?

name: sc_pcd_g2version
expr: sc_pcd_g2version > 0
for: 1m
labels:
  job: node
  severity: warning
annotations:
  message: Possibility of pcd g2version failure {{ $labels.instance }}.
  summary: pcd g2version Check failure


name: sc_pcd_encryption_server
expr: sc_pcd_encryption_server > 0
for: 1m
labels:
  job: node
  severity: warning
annotations:
  message: Possibility of **sc_pcd_g2version "25"** in encryption server failure {{ $labels.instance }}.
  summary: pcd encryption server Check failure

Solution

  • Generally, this is not possible.

    But in your exact case you could utilize a little trick: use sc_pcd_g2version > 0 and on(instance) sc_pcd_encryption_server > 0 instead of expression in your second alert.

    That way when using {{ $value }}, you get value of sc_pcd_g2version and not sc_pcd_encryption_server.

    Something like this:

    name: sc_pcd_encryption_server
    expr: sc_pcd_g2version > 0 and on(instance) sc_pcd_encryption_server > 0
    for: 1m
    labels:
      job: node
      severity: warning
    annotations:
      message: Possibility of {{ $value }} in encryption server failure {{ $labels.instance }}.
      summary: pcd encryption server Check failure
    

    This is possible because you don't use labels of sc_pcd_encryption_server, other than instance, and they will be same for both metrics I believe.

    Be sure to check that instance is enough to match to metrics. If not, add needed labels to on clause.