Does PromQL offer a way to filter metrics by comparing one label value to another label value for the same metric?
Like (hypothetical, is invalid syntax) some_metric{label_a!=label_b}. Or a way of saying some_metric{label_a} unless some_metric{label_b} that preserves labels on the left side.
I'm trying to find metrics by the relative value of two different labels on the same metric.
Consider some metric replication_lag with labels source_region and region. The source_region is where the service is pulling replication data from, the region is the service instance's location.
I want to find replication_lag only for cross region replication links, those where source_region != region. In other words I want to filter out those metrics where region == source_region.
Is there any way to do this in PromQL?
I've been unable to find any way by which a filter on one label's value can use the value of another label to perform a simple set-filtering operation.
Another way to do this would be to label-match two sets replication_lag{some_unique_key,region} and replication_lag{some_unique_key,source_region} - but this would require a means to label-match on an inequality condition for region and some_region i.e. perform an anti-join, which PromQL also does not appear to support.
The unless operator looks like it's close, but it requires exactly matching label sets. I need to preserve both region and source_region in the resulting set.
The best I've been able to come up with has been to use label rewriting and the unless operator to generate one label that can be used for set-subtraction.
some_metric{label_a!=label_b}
might be expressed as
label_join(some_metric{}, "join_key", ",", "label_a")
unless on (join_key)
label_join(some_metric{role="replica"}, "join_key", ",", "label_b")
It's verbose, cumbersome, probably inefficient, and it produces an unwanted join_key label that PromQL lacks a label_drop function to get rid of. There must be a better way.
Prometheus doesn't provide functions for comparing values across different labels the same metrics.
If you still need this functionality, you can try using an alternative Prometheus-like solution I work on, which supports such function - labels_equal. For example, the following MetricsQL query returns some_metric time series where region label doesn't equal source_region label:
some_metric unless labels_equal(some_metric, "region", "source_region")
This query uses unless operator for filtering out series with identical values for region and source_region labels.