prometheusprometheus-node-exporter

How to set up custom metrics_path per target?


I have installed node_exporter in several of my servers. I want to add them all together into a main dashboard in Grafana. I grouped up all the targets under the same job_name so I can filter by this in Grafana.

In my prometheus.yml I have configured several targets. All of them are node_exporter/metrics clients:

scrape_configs:
  - job_name: node_exporter
    static_configs:
      - targets: ["nodeexporter.app1.example.com"]
      - targets: ["nodeexporter.app2.example.com"]
      - targets: ["nodeexporter.app3.example.com"]
      - targets: ["nodeexporter.app4.example.com"]
    basic_auth:
      username: 'admin'
      password: 'my_password'

All works good because all these servers share the same default metrics_path and the same basic_auth.

Now I want to add a new target for the job node_exporter. But this one has a different path:

nodeexporter.app5.example.com/extra/metrics

I have tried to add it to the the static_configs but it doesn't work. I have tried:

static_configs:
  [... the other targets]
  - targets: ["nodeexporter.app5.example.com/extra/metrics"]

Also:

static_configs:
  [... the other targets]
  - targets: ["nodeexporter.app5.example.com"]
    __metrics_path__: "/extra/metrics"

Both return a YAML structure error.

How can I configure a custom metrics path for this new app?


Solution

  • The only solution I found is to divide the targets into 2 different scrap_configs/job_name. And setting the same label to both (so I can filter them in Grafana).

    scrape_configs:
      - job_name: node_exporter
        static_configs:
          - targets:
            - "nodeexporter.app1.example.com"
            - "nodeexporter.app2.example.com"
            - "nodeexporter.app3.example.com"
            - "nodeexporter.app4.example.com"
            labels:
              source: "node_exporter"
        basic_auth:
          username: "user"
          password: "my_password"
    
      - job_name: node_exporter_2
        metrics_path: "/extra/metrics"
        static_configs:
          - targets:
            - "nodeexporter.app5.example.com"
            labels:
              source: "node_exporter"
        basic_auth:
          username: "user_2"
          password: "my_password_2"
    

    In Grafana, instead of filtering by {job:"node_exporter"} I filter by {source:"node_exporter"}.