I'm setting up distributed tracing in my Spring Boot application using Micrometer Tracing and OpenTelemetry to send data to an OpenObserve instance, but I'm facing issues where traces are being recorded only intermittently.
Here's the setup: pom.xml Dependencies:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>com.squareup.okio</groupId>
<artifactId>okio</artifactId>
<version>3.9.1</version>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-logback-appender-1.0</artifactId>
<version>2.11.0-alpha</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp-logs</artifactId>
<version>1.26.0-alpha</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-logs</artifactId>
</dependency>
application.yml Configuration:
management:
otlp:
metrics:
export:
url: ${OPEN_OBSERVE_URL}/api/${OPEN_OBSERVE_ORG_NAME}/v1/metrics
enabled: true
step: 1m
headers:
enabled: true
Authorization: Basic ${OPEN_OBSERVE_PASSWORD}
organization: default
service:
name: default
tracing:
export:
enabled: true
sampling:
probability: 1.0
endpoint: ${OPEN_OBSERVE_URL}/api/${OPEN_OBSERVE_ORG_NAME}/v1/traces
headers:
active : true
enabled: true
Authorization: Basic ${TECHBD_OPEN_OBSERVE_PASSWORD}
organization: default
stream-name: ${OPEN_OBSERVE_STREAM_NAME}-traces
service:
name: default
The OPEN_OBSERVE_* environment variables are configured properly (double checked it), and I can verify them resolving in the final configurations by loging their values with the app startup. I'm using the logback configuration logback-appender-1.0 appender to properly integrate traces and spans with my existing logs using this property on the logback configuration file pattern %-X{traceId} (verified as well).
Despite these configurations, traces show up intermittently in OpenObserve.
I configured Spring Boot with Micrometer and OpenTelemetry, expecting consistent traces in OpenObserve. Instead, traces appeared intermittently.
The problem was with the configuration, setting it like this resolved the problem:
management:
tracing:
sampling:
probability: 1.0
By default, Spring Boot samples only 10% of requests to prevent overwhelming the trace backend. This property switches it to 100% so that every request is sent to the trace backend.