Exception in observability stack after upgrading Spring Boot from 3.1.3 to 3.2.0 using opentelemetry-java

Problem:

When upgrading the observability stack from Spring Boot version 3.1.3 to 3.2.0 using OTel Java SDK, the following exception is encountered:

2023-11-26T16:00:43.087Z  WARN [order service,,] 1 --- [order service] [BatchSpanProcessor_WorkerThread-1] [] i.o.exporter.zipkin.ZipkinSpanExporter   : Failed to export spans

org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "valid annotation timestamp required<EOL>"
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:103)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:183)
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:137)
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:932)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:881)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:781)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:663)
    at org.springframework.boot.actuate.autoconfigure.tracing.zipkin.ZipkinRestTemplateSender$RestTemplateHttpPostCall.doExecute(ZipkinRestTemplateSender.java:68)
    at org.springframework.boot.actuate.autoconfigure.tracing.zipkin.ZipkinRestTemplateSender$RestTemplateHttpPostCall.doEnqueue(ZipkinRestTemplateSender.java:75)
    at zipkin2.Call$Base.enqueue(Call.java:406)
    at io.opentelemetry.exporter.zipkin.ZipkinSpanExporter.export(ZipkinSpanExporter.java:82)
    at io.micrometer.tracing.otel.bridge.CompositeSpanExporter.lambda$export$3(CompositeSpanExporter.java:88)
    at java.base/java.lang.Iterable.forEach(Iterable.java:75)
    at io.micrometer.tracing.otel.bridge.CompositeSpanExporter.export(CompositeSpanExporter.java:88)
    at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.exportCurrentBatch(BatchSpanProcessor.java:327)
    at io.opentelemetry.sdk.trace.export.BatchSpanProcessor$Worker.run(BatchSpanProcessor.java:245)
    at java.base/java.lang.Thread.run(Thread.java:833)

Configuration:

Spring Boot Version: 3.2.0

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>3.2.0</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

Tracing Dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

Observations:

Encountering a HttpClientErrorException$BadRequest exception when using the observability stack in Spring Boot version 3.2.0. The issue does not occur with version 3.1.3.

Any insights or assistance on resolving this issue would be greatly appreciated.

Edit-1:

Reproduction Steps

• Clone the repository: food-delivery-backend
• Checkout the branch/tag: v3.2.0
• Check README.md on how to run
• Now check the order API logs in Grafana

Additional Information

• Working Sample (Spring Boot 3.2.0):
• v3.2.0/restaurant - No issues observed.
• Failing Sample (Spring Boot 3.2.0):
• v3.2.0/order - Exception occurs.

Edit-2:

• The error is not present when using Spring Boot version 3.1.3 for both brave and opentelemetry-java dependencies - Example
• The issue seems to be related to the interaction between Spring Boot 3.2.0 with opentelemetry-java

Edit-3:

Spring Boot 3.2.0 uses following tracing versions

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
    <version>1.2.0</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
    <version>1.31.0</version>
</dependency>

Spring Boot 3.1.3 uses following tracing versions

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
    <version>1.1.4</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
    <version>1.25.0</version>
</dependency>

I attempted different permutations of tracing versions with Spring Boot 3.2.0 to identify the specific dependency responsible for the exception. However, the issue persists with the following combinations:

1. Spring Boot 3.2.0 with Micrometer Tracing 1.2.0 and OTel Exporter Zipkin 1.25.0

Code: GitHub Link

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
  <version>1.2.0</version>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-zipkin</artifactId>
  <version>1.25.0</version>
</dependency>

2. Spring Boot 3.2.0 with Micrometer Tracing 1.1.4 and OTel Exporter Zipkin 1.31.0

Code: GitHub Link

<dependency>
 <groupId>io.micrometer</groupId>
 <artifactId>micrometer-tracing-bridge-otel</artifactId>
 <version>1.1.4</version>
</dependency>
<dependency>
 <groupId>io.opentelemetry</groupId>
 <artifactId>opentelemetry-exporter-zipkin</artifactId>
 <version>1.31.0</version>
</dependency>

In both cases, the exception persists. It remains unclear which specific combination of tracing versions with Spring Boot 3.2.0 triggers the issue.

Edit-4:

After implementing a minimal setup as suggested by @Jonatan Ivanov in the comments, I established two configurations—one utilizing an H2 database and the other incorporating the datasource-micrometer-spring-boot dependency for micrometer tracing on top of H2. Upon executing the application with the datasource-micrometer-spring-boot dependency, I encountered an error in the logs, indicating a potential issue with that specific dependency.

Below is the dependency configuration:

<dependency>
    <groupId>net.ttddyy.observation</groupId>
    <artifactId>datasource-micrometer-spring-boot</artifactId>
    <version>1.0.2</version>
</dependency>

Here are the corresponding GitHub links:

1. With only H2: GitHub Repository - Only H2

2. With H2 and datasource-micrometer-spring-boot: GitHub Repository - H2 and Micrometer Tracing

I have also raised a GitHub issue within the project, which can be found at: GitHub Issue - datasource-micrometer

Thanks to the collaboration on this, a bug was identified in Micrometer Tracing, fixed then released (1.2.1). Please upgrade your Micrometer Tracing version to use at least 1.2.1 (you can also wait till the next Boot release (3.2.1) since it should pick up the fixed version).