WebClient micrometer trace propagation is not working

Spring boot version 3.3.2

When using WebClient with each request to another microservice new traceId is created:

api-gateway service -> order-service

same traceID for api gateway and order service

order-service -> inventory-service (inventory-service has different traceId)

inventory service has different traceId

API Gateway config

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {
    @Bean
    public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
        http.csrf(ServerHttpSecurity.CsrfSpec::disable)
                .authorizeExchange(exchange -> exchange.pathMatchers("/eureka/**")
                        .permitAll()
                        .anyExchange().authenticated())

                .oauth2ResourceServer(jwt -> jwt.jwt(Customizer.withDefaults()));
        return http.build();
    }
}

Inventory service call (the method is annotated with @Observed)

InventoryResponse[] inventoryResponses;
inventoryResponses = webClientBuilder.build().get()
        .uri("http://inventory-service/api/inventory",
                uriBuilder -> uriBuilder.queryParams(multiValueMap).build())
        .retrieve()
        .bodyToMono(InventoryResponse[].class)
        .block();

Inventory service method

@Observed(name = "order.name",
        contextualName = "Inventory calls DB")
@Transactional(readOnly = true)
public List<InventoryResponse> isInStock(Map<String, String> skuCodes) {

    Map<String, Integer> skuCodesToInteger = new HashMap<>();
    for (Map.Entry<String, String> entry : skuCodes.entrySet()) {
        try {
            int value = Integer.parseInt(String.valueOf(entry.getValue()));
            skuCodesToInteger.put(entry.getKey(), value);

        } catch (NumberFormatException e) {
            log.error("Cannot cast {} to Integer", entry.getValue());
        }
    }

    log.info("Checking stock inventory");
    return inventoryRepository.findBySkuCodeIn(skuCodesToInteger.keySet()
                    .stream()
                    .toList())
            .stream()
            .map(inventory -> InventoryResponse
                    .builder()
                    .isInStock(inventory.getQuantity() >= skuCodesToInteger.get(inventory.getSkuCode()))
                    .skuCode(inventory.getSkuCode())
                    .build()).toList();
}

WebClient configuration class

@Configuration
public class WebClientConfig {

    @Bean
    @LoadBalanced
    public WebClient.Builder webClientBuilder() {
        return WebClient.builder();
    }
}

I have tried sending requests with RestTemplate and it worked:

same traceId for one api gateway call

RestTemplate configuration

@Configuration
public class RestTemplateConfiguration {

    @LoadBalanced
    @Bean
    public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {
        return restTemplateBuilder.build();
    }
}

Inventory service call (the method is annotated with @Observed)

URI uri = UriComponentsBuilder
    .fromUriString("http://inventory-service/api/inventory")
    .queryParams(multiValueMap).build().toUri();

InventoryResponse[] inventoryResponses = restTemplate
    .getForObject(uri, InventoryResponse[].class);

Is it vital to stay with webClient in this case, or I can use restTemplate? What is the cause of issue?

Updated Thanks to @brian-clozel for the hint. After I updated my webClient bean:

public WebClient webClient(WebClient.Builder webClientBuilder) {
    return webClientBuilder
        .filter(lbFunction)
        .build();
}

I got the correct traceId across the whole request, but I had to change my url user in order-service from dynamic address to static, e.g. from http://inventory-service/api/inventory to http://localhost:8082/api/inventory. Otherwise I would get an error:

2024-08-04T20:25:25.651+03:00 ERROR 1019862 --- [order-service] [nio-8081-exec-2] [order-service,,]o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] threw exception io.netty.resolver.dns.DnsErrorCauseException: Query failed with SERVFAIL
    at io.netty.resolver.dns.DnsResolveContext.onResponse(..)(Unknown Source) ~[netty-resolver-dns-4.1.111.Final.jar:4.1.111.Final]

2024-08-04T20:25:25.654+03:00 ERROR 1019862 --- [order-service] [nio-8081-exec-2] [order-service,,]o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.web.reactive.function.client.WebClientRequestException: Failed to resolve 'inventory-service' [A(1), AAAA(28)] after 2 queries ] with root cause io.netty.resolver.dns.DnsErrorCauseException: Query failed with SERVFAIL
    at io.netty.resolver.dns.DnsResolveContext.onResponse(..)(Unknown Source) ~[netty-resolver-dns-4.1.111.Final.jar:4.1.111.Final]

And to solve this problem (and use my server-service name in url) I added a load balancer filter to my webClient. So eventually this is my webClient config:

@Configuration
@Slf4j
@RequiredArgsConstructor
public class WebClientConfig {
    private final ReactorLoadBalancerExchangeFilterFunction lbFunction;

    @Bean
    public WebClient webClient(WebClient.Builder webClientBuilder) {
        return webClientBuilder
                .filter(lbFunction)
                .build();
    }
}

Just like you've used RestTemplateBuilder to create your client, you will need to use the auto-configured WebClient.Builder from Spring Boot if you want tracing support, as this builder is pre-configured with lots of things.

See the reference documentation about this.