asp.net-coreopen-telemetry

Duplicate open telemetry span ids when running multiple web applications in same process


I have two separate WebApplications running in same process, but they are listening on different ports. Each application configures OTEL like this (service name is unique to each app, for example there are "GATEWAY" and "SUBSCRIPTIONS" services:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
            tracerProviderBuilder
                .ConfigureResource(cfg =>
                {
                    cfg.AddService(serviceName);
                })
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddGrpcClientInstrumentation()
                .AddSource("GraphQL")
                .AddSource(GraphQLTelemetryProvider.SourceName)
                .AddSource(DiagnosticHeaders.DefaultListenerName)
                .AddSentry()
                .AddOtlpExporter(options =>
                {
                    options.Endpoint = new Uri("http://localhost:4317");
                })
    );

The problem is that all traces in Jaeger are duplicated - for example if there is http request to GATEWAY and then GATEWAY makes http request somewhere then in Jeager I see incoming HTTP request to both GATEWAY and SUBSCRIPTIONS on same "level" and then below GATEWAY I see outgoing http request from both GATEWAY and SUBSCRIPTIONS.

Also in Jaeger I see lot of "duplicate span IDs; skipping clock skew adjustment" warnings.

So in summary when span is "recorded" by service A then it's also recorded "by" all other services.

How I can make OTEL tracing for those services separate?


Solution

  • The issue you're hitting is because Activity is global to the process/AppDomain. So each TracerProvider you instantiate is listening to that activity and exporting it.

    Unfortunately this is expected behaviour in .NET and not specific to OpenTelemetry.