open-telemetrymicrometer-tracingopen-telemetry-java

Intermittent Traces with Micrometer, OpenTelemetry, and Spring Boot


I'm setting up distributed tracing in my Spring Boot application using Micrometer Tracing and OpenTelemetry to send data to an OpenObserve instance, but I'm facing issues where traces are being recorded only intermittently.

Here's the setup: pom.xml Dependencies:

<dependency> 
    <groupId>io.micrometer</groupId> 
    <artifactId>micrometer-registry-otlp</artifactId> 
</dependency>
<dependency> 
    <groupId>io.opentelemetry</groupId> 
    <artifactId>opentelemetry-exporter-otlp</artifactId> 
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency> 
    <groupId>com.squareup.okio</groupId> 
    <artifactId>okio</artifactId> 
    <version>3.9.1</version>
</dependency>       
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-logback-appender-1.0</artifactId>
    <version>2.11.0-alpha</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp-logs</artifactId>
    <version>1.26.0-alpha</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk-logs</artifactId>
</dependency>

application.yml Configuration:

management:
  otlp:
    metrics: 
      export: 
        url: ${OPEN_OBSERVE_URL}/api/${OPEN_OBSERVE_ORG_NAME}/v1/metrics
        enabled: true 
        step: 1m
        headers: 
          enabled: true 
          Authorization: Basic ${OPEN_OBSERVE_PASSWORD}
          organization: default
          service: 
            name: default
    tracing: 
      export:
        enabled: true
        sampling:
            probability: 1.0
      endpoint: ${OPEN_OBSERVE_URL}/api/${OPEN_OBSERVE_ORG_NAME}/v1/traces
      headers: 
        active : true
        enabled: true  
        Authorization: Basic ${TECHBD_OPEN_OBSERVE_PASSWORD}
        organization: default
        stream-name: ${OPEN_OBSERVE_STREAM_NAME}-traces
        service: 
          name: default

The OPEN_OBSERVE_* environment variables are configured properly (double checked it), and I can verify them resolving in the final configurations by loging their values with the app startup. I'm using the logback configuration logback-appender-1.0 appender to properly integrate traces and spans with my existing logs using this property on the logback configuration file pattern %-X{traceId} (verified as well).

Despite these configurations, traces show up intermittently in OpenObserve.

I configured Spring Boot with Micrometer and OpenTelemetry, expecting consistent traces in OpenObserve. Instead, traces appeared intermittently.


Solution

  • The problem was with the configuration, setting it like this resolved the problem:

    management:
      tracing:
        sampling:
          probability: 1.0
    

    By default, Spring Boot samples only 10% of requests to prevent overwhelming the trace backend. This property switches it to 100% so that every request is sent to the trace backend.