We have some java springboot applications running on docker containers in OpenShift.
We are running into an issue where Dynatrace open telemetry values are being injected into some parts of our logs.
Our Java projects have no mention or reference or configuration regarding Dynatrace, so this is injected completely from outside the jar(?, I honestly have no idea how it gets there), but long story short, when the message that is logged in the OpenShift console, and that is sent on an HTTP appender (since both the http appender and the console appender use the same JSON Layout) it is polluted with dynatrace parameters.
I was able to figure out that this only happens on the JSON Layout members that use a pattern resolver, but regardless if the pattern is calling the context or just using a static value, it still gets inserted.
Two of the members in the JSON Layout that use pattern, you can see they are very different, the nanoSecond using a static constant 0
"nanoOfSecond": {
"$resolver": "pattern",
"pattern": "0",
"stackTraceEnabled": false
},
"dumpanalysis": {
"$resolver": "pattern",
"pattern": "%replace{%throwable{separator(|)}}{\t}{ }",
"stackTraceEnabled": false
}
Resulting lines spit out on the Openshift log and attempted to be sent on the HTTP appender
"nanoOfSecond": "0 {dt.trace_id=692ebfa9aa8bda8eaceb24d681fe73e0, dt.span_id=007beacb18281124, dt.trace_sampled=true}"
"dumpanalysis": " {dt.trace_id=692ebfa9aa8bda8eaceb24d681fe73e0, dt.span_id=007beacb18281124, dt.trace_sampled=true}"
I've tried changing the pattern to replace the dt members with regex but that obviously didn't work, since this is also present on the nanoSecond that is a static constant. This value is being fed after the fact, at some point (w/e that point is)
I am at a complete loss here, any help is deeply appreciated.
Posting an answer here as this is what solved the problem, but both The_AM and Centic's answers are also correct, but I am not involved in the monitoring process as The_AM suggested.
Basically, the dynatrace telemetry was making my service fail when the log body was sent to the logging service as the controller was trying to map a String into a Long. What I did was adding a custom deserialzier to my model that would scrub the dynatrace data off before setting to the property, thus not needing to change the field type at all.
Yes, I understand this might remove data that the dynatrace analysis might need, but forcing it on a service to this level, to a point where it breaks the service itself seems wrong imo, aka: They broke it so they should fix it, if they don't I will fix it myself, if they don't like my fix, they should fix it.