open-telemetrydatadogopen-telemetry-javaopen-telemetry-datadog

After replacing datadog java agent with the opentelemetry java agent, metrics are not flowing to the datadog backend

We have datadog setup in our organisation for monitoring.
We use datadog to check the metrics and view the exception. We don't use it to see the logs.

We are planning to replace the datadog javaagent with opentelementry javaagent(due to better instrumentation support for vert.x) but keep our metrics that is flowing to the datadog backend intact.

We are following the documentation https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent/?tab=host

enter image description here

Our configuration for opentelementry java agent(Version 2.3.0) is

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://<datadog agent ip>:4318/v1/logs
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<datadog agent ip>:4318/v1/metrics
OTEL_EXPORTER_OTLP_ENDPOINT=http://<datadog agent ip>:4318/v1/{signal}
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<datadog agent ip>:4318/v1/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=supply

Datadog agent version - 7.51.0 and configuration

## OTLP ingest related configuration
  otlp:
    receiver:
      protocols:
        # datadog.otlp.receiver.protocols.grpc - OTLP/gRPC configuration
        grpc:
          # datadog.otlp.receiver.protocols.grpc.enabled -- Enable the OTLP/gRPC endpoint
          enabled: false
          # datadog.otlp.receiver.protocols.grpc.endpoint -- OTLP/gRPC endpoint
          endpoint: "0.0.0.0:4317"
          # datadog.otlp.receiver.protocols.grpc.useHostPort -- Enable the Host Port for the OTLP/gRPC endpoint
          useHostPort: true

        # datadog.otlp.receiver.protocols.http - OTLP/HTTP configuration
        http:
          # datadog.otlp.receiver.protocols.http.enabled -- Enable the OTLP/HTTP endpoint
          enabled: true
          # datadog.otlp.receiver.protocols.http.endpoint -- OTLP/HTTP endpoint
          endpoint: "0.0.0.0:4318"
          # datadog.otlp.receiver.protocols.http.useHostPort -- Enable the Host Port for the OTLP/HTTP endpoint
          useHostPort: true
    logs:
      # datadog.otlp.logs.enabled -- Enable logs support in the OTLP ingest endpoint
      enabled: false

Goal -> We want to send the metrics to the datadog backend using the datadog agent by the opentelemetry instrumentation(java agent) as shown in the image above.

We are able to access the datadog agent http://:4318/v1/metrics from the service pod where we added the open telementry instrumentation.

We are unable to view the metrics for the service in the datadog UI for the service instrumented with opentelementry java agent.

We are seeing one exception in that service pod logs file

[otel.javaagent 2024-05-14 16:10:24:760 +0530] [OkHttp http://172.31.22.114:4318/...] WARN io.opentelemetry.exporter.internal.http.HttpExporter - 
Failed to export logs. Server responded with HTTP status code 404. Error message: Unable to parse response body, HTTP status message: Not Found
Solution

  • Everything seems to be properly configured, except your environment variables.

    You have to either use the OTEL_EXPORTER_OTLP_<SIGNAL>_ENDPOINT with the full path, or OTEL_EXPORTER_OTLP_ENDPOINT just till port.

    Eg.:

    OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://<datadog agent ip>:4318/v1/logs
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://<datadog agent ip>:4318/v1/metrics
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<datadog agent ip>:4318/v1/traces

    or

    OTEL_EXPORTER_OTLP_ENDPOINT=http://<datadog agent ip>:4318

    The OTel Agent will append /v1/<signal> automatically when using the 2nd approach.

    Also, you can configure the OTel agent to suppress a signal if you do not want to send it anywhere, using the following environment variables:

    OTEL_<SIGNAL>_EXPORTER=none