prometheuspromql

how to list all the missing series though the count_over_time over a condition is not met


When I run the below, I could see 3 series (out of 5 series in total) which satisfy > 500 condition, but how can I list other 2 series also as 0?

count_over_time((some_metric{env="test"} > 500 )[10m:15s])

3 series screenshot

I expect all 5 series


Solution

  • If you want adding filtered out time series with zeroed values to query results, then add or q*0 to the end of the query as suggested in the comment to the question:

    count_over_time((some_metric{env="test"} > 500)[10m:15s])
      or
    some_metric{env="test"}*0
    

    This query uses or operator for adding filtered out series to query results. Multiplying by 0 is needed for replacing the real series values with zeroes.

    Note that count_over_time() may return unexpected results when a subquery filter is used inside it.

    For example, it may skip counting raw samples if the step value inside square brackets (15s in the query above) exceeds the interval between raw samples (aka scrape interval).

    It may also count the same sample multiple times if the step value is smaller than the actual scrape interval.

    P.S. If you want counting the exact number of samples over some specified time range with values greater than some specified value, then you can use count_gt_over_time function from MetricsQL (I'm the core developer of MetricsQL). For example, the following query returns the exact number of raw samples for time series matching some_metric{env="test"} filter with values exceeding 500 over the last 10 minutes:

    count_gt_over_time(some_metric{env="test"}[10m], 500)