How to bridge OpenTelemetry (opentelemetry-java-instrumentation) and Micrometer

My set-up:

1. Spring boot app based on version 3.1.3
2. Added dependency spring-boot-starter-actuator
3. Added custom metric MyHealthMetricsExportConfiguration according to the documentation (shown below)
4. Under /actuator/metrics I can see health metric
5. I've downloaded the latest otel java agent from github
6. I've added the config in intellij "vm options":

-javaagent:./opentelemetry-javaagent.jar
-Dotel.metrics.exporter=otlp

1. I've run otel collector with receiver set to otlp

The question

I don't know how to make otel agent 'to see' and export spring metrics like f.e. 'MyHealthMetricsExportConfiguration'. On that blog it should be as simple as:

  // Unregister the OpenTelemetryMeterRegistry from Metrics.globalRegistry and make it available
  // as a Spring bean instead.
  @Bean
  @ConditionalOnClass(name = "io.opentelemetry.javaagent.OpenTelemetryAgent")
  public MeterRegistry otelRegistry() {
    Optional<MeterRegistry> otelRegistry = Metrics.globalRegistry.getRegistries().stream()
        .filter(r -> r.getClass().getName().contains("OpenTelemetryMeterRegistry"))
        .findAny();
    otelRegistry.ifPresent(Metrics.globalRegistry::remove);
    return otelRegistry.orElse(null);
  }

But that doesn't work for me. First of all durig bean creation Metrics.globalRegistry.getRegistries() is empty. Second of all, no health metric is available in otel collector. What can I do to make this agent send my custom metric to collector ?

---

@Configuration(proxyBeanMethods = false)
public class MyHealthMetricsExportConfiguration {

    public MyHealthMetricsExportConfiguration(MeterRegistry registry, HealthEndpoint healthEndpoint) {
        // This example presumes common tags (such as the app) are applied elsewhere
        Gauge.builder("health", healthEndpoint, this::getStatusCode).strongReference(true).register(registry);
    }

    private int getStatusCode(HealthEndpoint health) {
        Status status = health.health().getStatus();
        if (Status.UP.equals(status)) {
            return 3;
        }
        if (Status.OUT_OF_SERVICE.equals(status)) {
            return 2;
        }
        if (Status.DOWN.equals(status)) {
            return 1;
        }
        return 0;
    }

}

You need to enable OpenTelemetry bridge for Micrometer. It's disabled by default since version 2.0.0 of OpenTelemetry agent: https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0

Agent Usage

For agent usage (your case), add this system property to your VM options: -Dotel.instrumentation.micrometer.enabled=true.

And I'd recommend setting the console exporter first (-Dotel.metrics.exporter=console), to verify just in console what gets exported regardless of what happens on the receiving side (on the collector).

P.S. You don't need the code piece from that blog post you mentioned (public MeterRegistry otelRegistry() {...). Maybe it is/was useful for something, but it's 2 years old and OpenTelemetry is still in alpha and has lots of breaking changes. So as of today, it worked without it for me. Note: the OpenTelemetryMeterRegistry that they mention there, is coming from the OpenTelemetry bridge lib (it's included in both agent and spring starter automatically).

OpenTelemetry Spring Starter Usage

For anybody using opentelemetry spring starter (io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter) instead of the agent, the micrometer bridge is already included into it, so enable it by setting the same system property (for example, in application.properties, add line otel.instrumentation.micrometer.enabled=true).

---

I tried both options with the latest version (2.3.0), it worked (Spring Boot 3.2.5). Using your class, I got the following metric log (among many others) in the console output (see ...name=health...value=3.0...):

[otel.javaagent 2024-05-11 00:11:00:926 +0400] [PeriodicMetricReader-1] INFO io.opentelemetry.exporter.logging.LoggingMetricExporter - metric: ImmutableMetricData{resource=Resource{schemaUrl=https://opentelemetry.io/schemas/1.24.0, attributes={host.arch="amd64", host.name="DESKTOP-NCP", os.description="Windows 10 10.0", os.type="windows", process.command_line="C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:13011,suspend=y,server=n -javaagent:./opentelemetry-javaagent.jar -Dotel.instrumentation.micrometer.enabled=true -Dotel.metrics.exporter=console -XX:TieredStopAtLevel=1 -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true -Dmanagement.endpoints.jmx.exposure.include=* -javaagent:C:\Users\Artem\AppData\Local\JetBrains\IntelliJIdea2024.1\captureAgent\debugger-agent.jar -Dfile.encoding=UTF-8 com.backend.BackendApplication", process.executable.path="C:\Program Files\Eclipse Adoptium\jdk-17.0.10.7-hotspot\bin\java.exe", process.pid=44220, process.runtime.description="Eclipse Adoptium OpenJDK 64-Bit Server VM 17.0.10+7", process.runtime.name="OpenJDK Runtime Environment", process.runtime.version="17.0.10+7", service.instance.id="3c1b88b9-bdf6-457f-9e46-abb511930f80", service.name="backend", telemetry.distro.name="opentelemetry-java-instrumentation", telemetry.distro.version="2.3.0", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.37.0"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.micrometer-1.5, version=null, schemaUrl=null, attributes={}}, name=health, description=, unit=, type=DOUBLE_GAUGE, data=ImmutableGaugeData{points=[ImmutableDoublePointData{startEpochNanos=1715371800406007400, epochNanos=1715371860420845000, attributes={}, value=3.0, exemplars=[]}]}}

---

Separately, Micrometer bridge is available on this page (find it in the table): https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#libraries--frameworks

---

Watch out: enabling Micrometer bridge will also start exporting its default Spring metrics and some have the same names as OpenTelemetry instrumentations. Because they have different attributes, the collector treats them as the same metric under different attributes, it seems. So in Grafana, for example, you'll see a single jvm_memory_used metric, but some dimensions will be duplicated (G1 Eden Space duplicate example):

...
comes from "io.opentelemetry.micrometer-1.5" bridge:
jvm_memory_used{area="heap", id="G1 Eden Space", instance="07d3f12f-c3a5-43bf-a268-bb80bceb6c56", job="stolpy-backend"}
...
comes from "io.opentelemetry.runtime-telemetry-java8":
jvm_memory_used{instance="07d3f12f-c3a5-43bf-a268-bb80bceb6c56", job="stolpy-backend", jvm_memory_pool_name="G1 Eden Space", jvm_memory_type="heap"}

So you probably want to disable default Spring Micrometer metrics, it should be explained somewhere in the docs how to do that. This SO post for example mentions something: Spring Boot Actuator/Micrometer Metrics Disable Some