prometheusprometheus-alertmanager

How to display values of metrics in alert rules prometheus


I create a rule in prometheus alertmanager, which tell about least space on mountpoints in percents - and additional to this i want to show how much least space in gigabytes , but i do not want to hardcoded a mountpoint to show gigabytes, i want use $labels.mountpoint which come from a "expr" , not hardcode.

i find a similar problem on this link https://github.com/prometheus/alertmanager/issues/549 but in this case using hardcoded mountpoint

this is my rule

- alert: OutOfDiskSpace
    expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
    for: 1m
    labels:
      severity: Critical
    annotations:
      description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf `node_filesystem_avail_bytes / 1024 / 1024 / 1024` | query | first | value | humanize }}"

when i used node_filesystem_avail_bytes / 1024 / 1024 / 1024 in VALUE, i do not take the mountpoint from expression, but i know where actual value, - it's in $labels.mountpoint which i can't use in template or don't know how to do this


Solution

  •   - alert: OutOfDiskSpace
          expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
          for: 5s
          labels:
            severity: Critical
          annotations:
            description: "Disk is almost full (< 10% left)\n {{ $labels.instance_short }}\n {{ $labels.mountpoint }}\n VALUE = {{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint | query | first | value | humanize1024 }}"