prometheusdistributed-computingdistributed-systemmicrometer

Micrometer & Prometheus with Java subprocesses that can't expose HTTP


I have a complex product that runs like this.

  1. A parent Java process which expose an HTTP service.
  2. The parent process starts worker subprocesses (new JVM) and manage the lifecycle of them.
  3. Worker processes can't have HTTP server (for security and many other reasons), they communicate with other components through TCP connections. A parent process can manage many worker processes for a very long period of time.
  4. We need to use Prometheus to scrape metrics.
  5. Both parent and worker processes have micrometer observation instruments, but only parent exposed an HTTP server.
  6. We understand the limitations of Prometheus push gateway so we decided not to use it.

Question 1: How do we config Prometheus to scrape metrics from the workers when there is no HTTP.

Question 2: If Q1 is not doable, is there a way that we can send all metrics from workers to parent and let the parent to export these metrics? How should this work? Is there existing Micrometer tools to send a whole registry to another process through TCP? Once the parent received the registry from worker, how does the parent merge it with its own metrics?

I tried to find some examples but everything online shows that each JVM should have its own HTTP endpoint for scraping.


Solution

  • How do we config Prometheus to scrape metrics from the workers when there is no HTTP.

    Check out https://github.com/micrometer-metrics/prometheus-rsocket-proxy it has a server and a client component. You run the server in your infra and add the client to your app. The client connects to the server and sends metrics data to it. The Prometheus backend scrapes the server not the app (no HTTP needed).