restcurlprometheusprometheus-alertmanagerprometheus-java

How to get all metric names from Prometheus server filtered by a particular label


We want to get all metric names from Prometheus server filtered by a particular label.

Step 1 : Used following query to get all metric names, query succeeded with all metric names.

curl -g 'http://localhost:9090/api/v1/label/__name__/values

Step 2 : Used following query to get all metrics names filtered by label, but query still returned all metric names.

curl -g 'http://localhost:9090/api/v1/label/__name__/values?match[]={job!="prometheus"}'

Can somebody please help me filter all metric names by label over http? Thanks

curl -G -XGET http://localhost:9090/api/v1/label/__name__/values --data-urlencode 'match[]={__name__=~".+", job!="prometheus"}'

@anemyte, Still returns all the results. Can you please check the query


Solution

  • Although this seems simple at the first glance, it turned out to be a very tricky thing to do.

    1. The match[] parameter and its value have to be encoded. curl can do that with --data-urlencode argument.

    2. The encoded match[] parameter must be present in the URL and not in application/x-www-form-urlencoded header (where curl puts the encoded value by default). Thus, the -G (the capital one!) key is also required.

    3. {job!="prometheus"} isn't a valid query. It gives the following error:

      parse error: vector selector must contain at least one non-empty matcher

      It is possible to overcome with this inefficient regex selector: {__name__=~".+", job!="prometheus"}. It would be better to replace it with another selector if possible (like {job="foo"}, for example).

    Putting all together:

    curl -XGET -G 'http://localhost:9090/api/v1/label/__name__/values' \
      --data-urlencode 'match[]={__name__=~".+", job!="prometheus"}' 
    

    Using selectors as in the example above became possible since Prometheus release v2.24.0.