I'm working on my local computer trying to get Jaeger to show OpenTelemetry traces from a .NET 8 ASP.NET Core project.
Jaeger 1.58 is running on Docker-desktop and setup with the command: docker run -d --name jaeger -e COLLECTOR_OTLP_ENABLED=true -p 16686:16686 -p 4317:4317 -p 4318:4318 jaegertracing/all-in-one
The Jaeger logs show the OTLP collector starting up (and I can access the UI):
2024-07-10 11:33:02 {"level":"info","ts":1720625582.0473268,"caller":"otlpreceiver@v0.104.0/otlp.go:102","msg":"Starting GRPC server","endpoint":"localhost:4317"}
2024-07-10 11:33:02 {"level":"info","ts":1720625582.0492482,"caller":"otlpreceiver@v0.104.0/otlp.go:152","msg":"Starting HTTP server","endpoint":"localhost:4318"}
My .NET is using the OtlpExporter to connect to http://localhost:4317 with the OtlpExportProtocol.Grpc protocol.
I've used OTEL_DIAGNOSTICS.json to enable logging from the OtlpExporter and I'm getting the following error back:
2024-07-10T16:16:45.7292747Z:Exporter failed send data to collector to {0} endpoint. Data will not be sent. Exception: {1}{http://localhost:4317/}{Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. (InvalidResponse) HttpIOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. (InvalidResponse) HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)", DebugException="System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. (InvalidResponse)")
---> System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. (InvalidResponse)
---> System.Net.Http.HttpIOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. (InvalidResponse)
---> System.Net.Http.HttpIOException: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)
at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|61_1()
at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.SendDataAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.StreamExtensions.WriteMessageAsync[TMessage](Stream stream, GrpcCall call, TMessage message, Action`2 serializer, CallOptions callOptions)
at Grpc.Net.Client.Internal.PushUnaryContent`2.WriteMessageCore(Task writeMessageTask)
at System.Net.Http.Http2Connection.Http2Stream.SendRequestBodyAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageInvoker.<SendAsync>g__SendAsyncWithTelemetry|6_0(HttpMessageHandler handler, HttpRequestMessage request, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
--- End of inner exception stack trace ---
at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseHeadersCoreAsync()
at Grpc.Net.Client.Internal.HttpClientCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at Grpc.Core.Interceptors.InterceptingCallInvoker.<BlockingUnaryCall>b__3_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
at Grpc.Core.ClientBase.ClientBaseConfiguration.ClientBaseConfigurationInterceptor.BlockingUnaryCall[TRequest,TResponse](TRequest request, ClientInterceptorContext`2 context, BlockingUnaryCallContinuation`2 continuation)
at Grpc.Core.Interceptors.InterceptingCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
at OpenTelemetry.Proto.Collector.Logs.V1.LogsService.LogsServiceClient.Export(ExportLogsServiceRequest request, CallOptions options)
at OpenTelemetry.Proto.Collector.Logs.V1.LogsService.LogsServiceClient.Export(ExportLogsServiceRequest request, Metadata headers, Nullable`1 deadline, CancellationToken cancellationToken)
at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.OtlpGrpcLogExportClient.SendExportRequest(ExportLogsServiceRequest request, DateTime deadlineUtc, CancellationToken cancellationToken)}
I tried using the below code to allow unsecured gRPC but the error is still the same: AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
"msg":"Starting GRPC server","endpoint":"localhost:4317"
This may be due to recent change in OTEL Collector where receivers started defaulting to localhost instead of 0.0.0.0 (all IPs). If you run Jaeger in a container then localhost means it may not be listening to anything outside of the container itself. This issue affects Jaeger v1.59 (https://github.com/jaegertracing/jaeger/issues/5737) and is fixed in the next upcoming release where we go back to listening on all IPs (https://github.com/jaegertracing/jaeger/pull/5739). As a workaround you can instruct Jaeger to listen on all IPs manually by passing these env vars:
COLLECTOR_OTLP_GRPC_HOST_PORT=0.0.0.0:4317
COLLECTOR_OTLP_HTTP_HOST_PORT=0.0.0.0:4318