.netopen-telemetryopen-telemetry-collector

How to configure exceptions telmetry with .net opentelemetry sdk to work with otel collector and azuremonitor exporter


I am designing observability solution for my .net project. My aim is to send telemetry data to open telemetry collector and then primarily export them to azure monitor (app insights).

In my .net solution, I am using OtlpExporter. Evrything works fine and i can see telemetry data in azure monitor. This is my setup in .net:

            var resourceBuilder = ResourceBuilder.CreateDefault()
                .AddService(
                    serviceName: otelConfig.ServiceName,
                    serviceNamespace: otelConfig.ServiceNamespace,
                    serviceVersion: otelConfig.ServiceVersion,
                    autoGenerateServiceInstanceId: false,
                    serviceInstanceId: otelConfig.ServiceInstanceId
                    ).AddTelemetrySdk();

            if (otelConfig.Instrumentation.Logs)
            {
                services.AddLogging(loggingBuilder =>
                {
                    loggingBuilder.AddOpenTelemetry(options =>
                    {
                        options.IncludeFormattedMessage = true;
                        options.IncludeScopes = true;
                        options.SetResourceBuilder(resourceBuilder);

                        otelConfig.AssignExporter(exporter => exporter.Otlp, otlpExporter =>
                        {
                            options.AddOtlpExporter(otlpOptions =>
                            {
                                otlpOptions.Protocol = otelConfig.GetProtocol();
                                otlpOptions.Endpoint = new Uri(otelConfig.Exporters.Otlp.Endpoint);
                            });
                        });

                    });
                });
            }

and this is config for otel collector:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  # HOTFIX: attribute proccessor for azure monitor ISSUE: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/29687
  attributes:
    actions:
      - key: http.status_code
        action: insert
        from_attribute: http.response.status_code
      - key: http.url
        action: insert
        from_attribute: url.full
      - key: http.target
        action: insert
        from_attribute: url.path
      - key: http.host
        action: insert
        from_attribute: server.address
      - key: http.scheme
        action: insert
        from_attribute: url.scheme
      - key: http.method
        action: insert
        from_attribute: http.request.method    
  batch:

exporters:
  file:
    path: logs/filename.json  
  logging:
    verbosity: detailed
  azuremonitor:
    # replace with valid conn string  
    connection_string: "InstrumentationKey=00000000-0000-0000-0000-000000000000IngestionEndpoint=https://westeurope-5.in.applicationinsights.azure.com/;LiveEndpoint=https://westeurope.livediagnostics.monitor.azure.com/;ApplicationId=00000000-0000-0000-0000-000000000000"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch,attributes]
      exporters: [file,azuremonitor]
    logs:
      receivers: [otlp]
      processors: [batch,attributes]
      exporters: [file,azuremonitor]

Only issue which i encountered is Exceptions. When I place this in my code:

var ex = new Exception("Test exception");

logger.LogError(ex, "My message");

It ends up in traces table in azure monitor.

When i export telemetry data directly from my .net app to azuremonitor, everything works fine.

My impression was, that .net open telemetry sdk should ensure correct semantics of open telemetry protocol. This should ensure that whether i use direct azure monitor exporter or otel collector, everything should be same.

As per documentation, exception type should be LogRecord with specific attributes. I can see this structure in my file log from otel collector. But still it wont end up in exceptions table in azure monitor.

Edit:

To clarify my question. I also tried recording exception on span which is described in doc. This is implemented in my .net project like so:

var activity = Activity.Current;

activity?.RecordException(ex);
            activity?.SetStatus(Status.Error.WithDescription(ex.Message));

This works correctly and is saved to exceptions table in azure monitor. My issue is, that i am working with legacy code with many _looger.LogError(ex,"Error occured") statements, which does not have any activity associated with it. But based on documentation, it SHOULD also be possible to log exception on LogRecord. Is there any workaround i could use in my .net code ? Alternatively is it possible to somehow transform LogRecord in otel collector ?


Solution

  • I ended up creating some kind of logger wrapper for open telemetry logger. This logger wrapper calls activity.AddEvent which adds Exception to Trace. This effectively means, that if there is Activity.Current, we log excpetion on it. When it is logged like this, it ends up in exceptions table.

    This approach enabled us, to have nice end-to-end trasaction view in app insights. But when exception is logged without Activity.Current it is logged in traces table.

    I created OtelExceptionLogger like so:

    public class OtelExceptionLogger : ILogger
    {
        private readonly ILogger _innerLogger;
    
        public OtelExceptionLogger(ILogger innerLogger)
        {
            _innerLogger = innerLogger;
        }
    
        public IDisposable BeginScope<TState>(TState state) => _innerLogger.BeginScope(state);
    
        public bool IsEnabled(LogLevel logLevel) => _innerLogger.IsEnabled(logLevel);
    
        public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
        {
            var activity = Activity.Current;
    
            if (exception is null || activity is null)
            {
                _innerLogger.Log(logLevel, eventId, state, exception, formatter);
    
                return;
            }
    
            activity.AddEvent(new ActivityEvent(
                "exception",
                tags: new ActivityTagsCollection{
                                { "exception.type", exception.GetType().ToString() },
                                { "exception.message", exception.Message },
                                { "exception.stacktrace", exception.StackTrace }
                    }
                ));
    
            activity.SetStatus(ActivityStatusCode.Error, exception.Message);
        }
    }
    

    Also you need LoggerProvider:

    public class OtelExceptionLoggerProvider : ILoggerProvider
    {
        private readonly ILoggerProvider _innerProvider;
    
        public OtelExceptionLoggerProvider(ILoggerProvider innerProvider)
        {
            _innerProvider = innerProvider;
        }
    
        public ILogger CreateLogger(string categoryName)
        {
            var innerLogger = _innerProvider.CreateLogger(categoryName);
            return new OtelExceptionLogger(innerLogger);
        }
    
        public void Dispose()
        {
            _innerProvider?.Dispose();
        }
    }
    

    Finally you have to replace loggingBuilder.AddOpenTelemetry with followingcode:

    services.AddLogging(loggingBuilder =>
    {
        var optionsMonitor = loggingBuilder.Services.BuildServiceProvider().GetService<IOptionsMonitor<OpenTelemetryLoggerOptions>>();
    
        var otelProvider = new OpenTelemetryLoggerProvider(optionsMonitor);
        
        if (otelConfig.Instrumentation.LogsAsExceptionEvents)
        {
            loggingBuilder.AddProvider(new OtelExceptionLoggerProvider(otelProvider));
        }
        else
        {
            loggingBuilder.AddProvider(otelProvider);
        }
    });