I have two similar metrics from the same job. It's the probe_success and probe_http_status_code metric of the Prometheus Blackbox Exporter. I'm also using Victoriametrics if that's of any help.
I now want to create a Grafana-panel that displays the probe_http_status_code of various entities but only if they have a probe_success == 0 in the given range.
I tried to receive both metrics at once taking advantage of the union-functionality of MetricsQL. To only receive the labels, that allow to connect both metrics, I'm using label_keep:
(label_keep(probe_success == 0, "instance"), label_keep(probe_http_status_code, "instance"))
Unfortunately the Status codes are discarded, as there are already the values of the probe_success-metric. Is there any way to combine them in a way that all Status Codes which correspond to a probe_success of 0 are discarded? Or maybe the value of probe_success may be transformed to a label to be lateron filtered in grafana itself?
To get list of all the received http codes for target that failed at least once within the range of your dashboard use query
probe_http_status_code and min_over_time(probe_success [$__range] @end() ) == 0
Here probe_http_status_code is returned only if min_over_time(probe_success [$__range] @end() ) == 0 also returned result with same labels. And the latter return result if within values of probe_success there was at least one zero (based on the fact that only possible values are 0 and 1).
Construct probe_success [$__range] @end() return range vector of length $__range (substituted by Grafana with actual range of dashboard) that ends on dashboard end time. @end() is used to pin this range selector to the end of dashboard for all values, otherwise range selector would return range vector relatively to the time of evaluated moment.