grafanagrafana-alerts

Grafana: Can't alert on a function metric?


I am sending two regular metrics to Grafana through statsD and plotting them as two series (A and B). The metrics are "total_users" and "unfound_users".

I am then creating a third series (C) which uses this expression:

asPercent(movingAverage(#A, '5min'),movingAverage(#B, '5min'))

Then I am attempting to create an alert on C that triggers when the max() of C over a 5 minute period is over 2. Essentially this would mean "trigger an alert when a 5-minute average of unfound_users is more than 2% of a 5-minute-average of total_users."

You can see that my derived series (C) is getting plotted properly, but the alerts do not seem to work at all. I'm just getting back empty series in the test response.

We are using Grafana v.6.1.6.

Here is the setup of the series: Setup of the series

The alert configuration: Alert configuration

And the empty test response: Empty test response

I'd appreciate some input! Thanks!


Solution

  • I figured out a workaround. Instead of using

    asPercent(movingAverage(#A, '5min'),movingAverage(#B, '5min')) 
    

    as my derived metric I needed to use the full paths in place of the reference ids.

    This worked:

    asPercent(movingAverage(stats.counters.stg.some.full.path.unfound_users.count, '5min'),
    movingAverage(stats.counters.stg.some.full.path.total_users.count, '5min'))