I'm facing a nightmarishly hard problem to solve for something that seems so trivial. Ultimately, I'm trying to detect a single increment of a counter metric in PromQL when there was no data previously in order to trigger an alert.
I've hit a dead-end in PromQL and the only other solution seems to be to initialise a counter to 0 using Benthos (i.e serve the counter as a value as 0, even though it's not been incremented in the processor pipeline). Is this possible? If it is, it would allow me to detect a change from 0 -> 1 instead of [no-data] -> 1.
Any guidance would be much appreciated.
One way around it is to use a mix of sequence and generate inputs:
input:
sequence:
inputs:
- generate:
count: 1
mapping: root = ""
processors:
- metric:
type: counter
name: test
labels:
foo: bar
- mapping: root = deleted()
- your_actual_input:
# ...
The message created by the generate input is used to set that test metric and then it's discarded.