I am designing observability solution for my .net project. My aim is to send telemetry data to open telemetry collector and then primarily export them to azure monitor (app insights).
In my .net solution, I am using OtlpExporter
. Evrything works fine and i can see telemetry data in azure monitor. This is my setup in .net:
var resourceBuilder = ResourceBuilder.CreateDefault()
.AddService(
serviceName: otelConfig.ServiceName,
serviceNamespace: otelConfig.ServiceNamespace,
serviceVersion: otelConfig.ServiceVersion,
autoGenerateServiceInstanceId: false,
serviceInstanceId: otelConfig.ServiceInstanceId
).AddTelemetrySdk();
if (otelConfig.Instrumentation.Logs)
{
services.AddLogging(loggingBuilder =>
{
loggingBuilder.AddOpenTelemetry(options =>
{
options.IncludeFormattedMessage = true;
options.IncludeScopes = true;
options.SetResourceBuilder(resourceBuilder);
otelConfig.AssignExporter(exporter => exporter.Otlp, otlpExporter =>
{
options.AddOtlpExporter(otlpOptions =>
{
otlpOptions.Protocol = otelConfig.GetProtocol();
otlpOptions.Endpoint = new Uri(otelConfig.Exporters.Otlp.Endpoint);
});
});
});
});
}
and this is config for otel collector:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
# HOTFIX: attribute proccessor for azure monitor ISSUE: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/29687
attributes:
actions:
- key: http.status_code
action: insert
from_attribute: http.response.status_code
- key: http.url
action: insert
from_attribute: url.full
- key: http.target
action: insert
from_attribute: url.path
- key: http.host
action: insert
from_attribute: server.address
- key: http.scheme
action: insert
from_attribute: url.scheme
- key: http.method
action: insert
from_attribute: http.request.method
batch:
exporters:
file:
path: logs/filename.json
logging:
verbosity: detailed
azuremonitor:
# replace with valid conn string
connection_string: "InstrumentationKey=00000000-0000-0000-0000-000000000000IngestionEndpoint=https://westeurope-5.in.applicationinsights.azure.com/;LiveEndpoint=https://westeurope.livediagnostics.monitor.azure.com/;ApplicationId=00000000-0000-0000-0000-000000000000"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch,attributes]
exporters: [file,azuremonitor]
logs:
receivers: [otlp]
processors: [batch,attributes]
exporters: [file,azuremonitor]
Only issue which i encountered is Exceptions
. When I place this in my code:
var ex = new Exception("Test exception");
logger.LogError(ex, "My message");
It ends up in traces
table in azure monitor.
When i export telemetry data directly from my .net app to azuremonitor, everything works fine.
My impression was, that .net open telemetry sdk should ensure correct semantics of open telemetry protocol. This should ensure that whether i use direct azure monitor exporter or otel collector, everything should be same.
As per documentation, exception type should be LogRecord
with specific attributes. I can see this structure in my file log from otel collector. But still it wont end up in exceptions
table in azure monitor.
Edit:
To clarify my question. I also tried recording exception on span which is described in doc. This is implemented in my .net
project like so:
var activity = Activity.Current;
activity?.RecordException(ex);
activity?.SetStatus(Status.Error.WithDescription(ex.Message));
This works correctly and is saved to exceptions
table in azure monitor. My issue is, that i am working with legacy code with many _looger.LogError(ex,"Error occured")
statements, which does not have any activity associated with it. But based on documentation, it SHOULD also be possible to log exception on LogRecord
. Is there any workaround i could use in my .net
code ? Alternatively is it possible to somehow transform LogRecord
in otel collector ?
I ended up creating some kind of logger wrapper for open telemetry logger. This logger wrapper calls activity.AddEvent
which adds Exception to Trace. This effectively means, that if there is Activity.Current
, we log excpetion on it. When it is logged like this, it ends up in exceptions table.
This approach enabled us, to have nice end-to-end trasaction view in app insights. But when exception is logged without Activity.Current
it is logged in traces
table.
I created OtelExceptionLogger
like so:
public class OtelExceptionLogger : ILogger
{
private readonly ILogger _innerLogger;
public OtelExceptionLogger(ILogger innerLogger)
{
_innerLogger = innerLogger;
}
public IDisposable BeginScope<TState>(TState state) => _innerLogger.BeginScope(state);
public bool IsEnabled(LogLevel logLevel) => _innerLogger.IsEnabled(logLevel);
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception exception, Func<TState, Exception, string> formatter)
{
var activity = Activity.Current;
if (exception is null || activity is null)
{
_innerLogger.Log(logLevel, eventId, state, exception, formatter);
return;
}
activity.AddEvent(new ActivityEvent(
"exception",
tags: new ActivityTagsCollection{
{ "exception.type", exception.GetType().ToString() },
{ "exception.message", exception.Message },
{ "exception.stacktrace", exception.StackTrace }
}
));
activity.SetStatus(ActivityStatusCode.Error, exception.Message);
}
}
Also you need LoggerProvider
:
public class OtelExceptionLoggerProvider : ILoggerProvider
{
private readonly ILoggerProvider _innerProvider;
public OtelExceptionLoggerProvider(ILoggerProvider innerProvider)
{
_innerProvider = innerProvider;
}
public ILogger CreateLogger(string categoryName)
{
var innerLogger = _innerProvider.CreateLogger(categoryName);
return new OtelExceptionLogger(innerLogger);
}
public void Dispose()
{
_innerProvider?.Dispose();
}
}
Finally you have to replace loggingBuilder.AddOpenTelemetry
with followingcode:
services.AddLogging(loggingBuilder =>
{
var optionsMonitor = loggingBuilder.Services.BuildServiceProvider().GetService<IOptionsMonitor<OpenTelemetryLoggerOptions>>();
var otelProvider = new OpenTelemetryLoggerProvider(optionsMonitor);
if (otelConfig.Instrumentation.LogsAsExceptionEvents)
{
loggingBuilder.AddProvider(new OtelExceptionLoggerProvider(otelProvider));
}
else
{
loggingBuilder.AddProvider(otelProvider);
}
});