prometheusgrafanagrafana-alerts

How to create a Grafana alert when a prometheus gauge value doesn't change for a long time?


I have a custom gauge metric (say, my-custom-metric) that I created using Prometheus client in my backend. This metric's value is updated periodically using a cron job. It represents the number of operations (of a specific type) performed by the backend i.e. the value of metric always increases.

I want to create a Grafana alert if the value of my-custom-metric is constant for a long time (say 1 hour). How will the query for this look like?


Solution

  • I would make use of the CHANGES prometheus function:

    changes(my-custom-metric[1h]) == 0
    

    changes() function calculates the number of times the value of my-custom-metric has changed over the past hour. The changes() function returns the number of times the value has changed within the given time range, so if the value has remained constant, the result will be 0. The == 0 comparison at the end of the query checks if the number of changes is equal to 0, which means the value has remained constant.