springspring-bootprometheusmicrometermicrometer-tracing

Disable exemplars support in Spring Boot 3.2 to avoid Prometheus problem with scrapping metrics


New Spring Boot added exemplars support: https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.2-Release-Notes#broader-exemplar-support-in-micrometer-112

After upgrading to Spring Boot 3.2.0 Prometheus stopped scrapping a metrics from our apps, because it requires a Prometheus version 2.43.0 or newer:

metric name web_client_requests_seconds_count does not support exemplars

I think there should be a way to disable this until we won't upgrade a Prometheus.


Solution

  • I think this is because you are using an unsupported version of Prometheus server. You will need this change in Prometheus: Exemplars support for all time series #11982. It was introduced in Prometheus 2.43.0. Also see discussion about this: Add exemplar support to _count #3996.

    According to the Prometheus support terms, you should use at least 2.45.

    If you really want to hack this (please don't and use a supported version of Prometheus instead), I think if you create a SpanContextSupplier @Bean that always says isSampled false and it can return null for the spanId and traceId, you will not see those exemplars.

    It is also possible to modify the value of the Accept header and use text/plain instead of application/openmetrics-text in a custom controller or in a gateway or reverse-proxy. This should work because the OpenMetrics format supports exemplars while the Prometheus format does not (Prometheus asks for the OpenMetrics format by default).

    Update

    In Spring Boot 3.3.0 and Micrometer 1.13.0, we added support to the Prometheus 1.x Java Client which supports conditionally enabling exemplars on all time series (i.e.: _count), see PrometheusConfig in Micrometer.