I'm using prometheus + grafana for collecting and displaying metrics. If one of my devices stops reporting grafana will continue to graph the last value. For example, if one of my devices reported its temperature 2 days ago and the temperature was 50 degrees, grafana will display a flat line for 50 degrees for the last 2 days. I would expect this timeseries graph to dip down to zero if the device wasn't reporting any metrics.
Is this just the way prometheus and grafana behave or am I missing something in the configuration? I would like this timeseries graph to chart zeros if a device is not reporting rather than showing a flat line of its last value.
Additional info:
Using prometheus.net to push metrics to prometheus' push gateway.
I've tried using the Null value: null as zero option under the Stacking and Null Value section of the graph in grafana.
This is the expected behavior of push gateway. Once a value is pushed, it will stay forever. This is assumed by the dev team and TTL or expiration of metrics have been refused
The usual solution to this problem is to push a Unix timestamp of the measurement and have a rule alerting if it becomes too old or use a UNLESS time() - timestamp_metric condition.