Prometheus: filter query based on another metric

Say I have two metrics in prometheus, both counters:

• requests_processed_total
• requests_failed_total

They both have a matching service label. Example:

requests_processed_total{service="news"} 1097
requests_processed_total{service="store"} 487
requests_failed_total{service="news"} 23
requests_failed_total{service="store"} 89

How to query the requests_failed_total, but only for services whose request_processed_total > 1000.

I'm expecting the following response:

requests_failed_total{service="news"} 23

# Note that the "store" service is excluded

If you are using Grafana you can do the following:

(1) Create a dashboard

(2) Click on Dashboard settings > Variables > New

(3) Create a variable with the following:

Name        = service
Type        = Query

Data source = Prometheus
Query       = query_result(request_processed_total>5)
Regex       = /service="(.*)"/

(4) Use the "service" variable to show the "requests_failed_total" metrics in any panel (you can also use the "repeat for " Grafana feature.