prometheuspromqlprometheus-operator

Prometheus metric for over time not working


I need to create a prometheus query that bring the numbers of event in one hour and / on day

when run the following and run request I see that the number increased as expected , 5 api calls the counter raised in 5.

this is the basic query vector_component_sent_events_total

Now I tried the following to get the number of requests for one hour but I get wrong numbers (like 0.0.5 which is not the number of request that I send in 1 hour...)

I tried the following, what am I missing here?

sum(rate(vector_component_sent_events_total{component_id="ta", job="nt"}[1h]))

I got something like 0.004201679495327872 , but I send only 5 (or bit more) request in hour ... any idea?

https://vector.dev/docs/administration/monitoring/


Solution

  • If you want an increase in your metric for 1 hour you could use this:

    increase(vector_component_sent_events_total{component_id="ta", job="nt"}[1h])
    

    Notice also, this clause in official documentation of increase function:

    The increase is extrapolated to cover the full time range as specified in the range vector selector, so that it is possible to get a non-integer result even if a counter increases only by integer increments.

    If you want an exact integer increase, you can either use round on the result of increase, or use some workarounds.

    One such workaround is using subtraction with offset operator

    vector_component_sent_events_total{component_id="ta", job="nt"}
     - vector_component_sent_events_total{component_id="ta", job="nt"} offset 1h
    

    this has a significant drawback: violently incorrect results near counter resets.

    A bit more correct result near resets might produce query

    vector_component_sent_events_total{component_id="ta", job="nt"}
     - vector_component_sent_events_total{component_id="ta", job="nt"} offset 1h >= 0
    or round(increase(vector_component_sent_events_total{component_id="ta", job="nt"}[1h]))
    

    It will count "increase" based on subtraction outside of resets, and fallback to rounding of increase call near resets.

    CAUTION: Please remember, this is a hack and it could produce incorrect data in some circumstances.