bosun

Using Bosun, how can I make an alert not trigger for certain times of day?

I'm using bosun to monitor a metric that should be non-zero most of the working day, but is ok to be zero/unavailable during the night.

alert myalert {
    $notes = `This alert triggers when cw- orders haven't been received recently.`
    template = noweborders
    unknownIsNormal = 0
    $metricLabel = Orders
    $metric = q("max:1d-max:rate{counter,,1}:metricname{filtercategory=cw-,host=*}", "2w", "")
    $graph = q("max:1m-max:rate{counter,,1}:metricname{filtercategory=literal_or(cw-),host=wildcard(*)}", "1d", "")
    $uptimeStoppedWarn = since($metric) > d("2h")
    $uptimeStoppedCrit = since($metric) > d("4h")
    $lastOrder = ungroup(since($metric)) / 60 / 60
    warn = $uptimeStoppedWarn
    crit = $uptimeStoppedCrit
    warnNotification = georgeemail
    critNotification = georgeemail
}

How can I best adapt this alert so if the metric would be zero or unknown between the hours of say, 8pm and 8am it wouldn't trigger the alert? I've looked through the documentation, but I'm not sure how to do queries relating to time of day.

Solution

  • If you're really not interested whether or not your alert condition is met during certain times, you could introduce an additional condition to your alerts and make use of the epoch() function. epoch() returns the current timestamp at the evaluation of the alerts. Then add that to the crit or warn condition. Something like this will work:

    alert myalert {
    $notes = `This alert triggers when cw- orders haven't been received recently.`

    (...)

    $eight_pm = 20*60*60
    $eight_am = 8*60*60
    $seconds_today = epoch() % 86400
    $is_before_8am = $seconds_today < $eight_am
    $is_after_8pm = $eight_pm <= $seconds_today
    $should_alert = !$is_before_8am && !$is_after_8pm

    warn = $uptimeStoppedWarn && $should_alert
    crit = $uptimeStoppedCrit && $should_alert
    warnNotification = georgeemail
    critNotification = georgeemail
}

    This will prevent the alert from going into the critical or warning state outside of 8am to 8pm. Might be worth extracting that into a macro if you're using that across multiple alerts.