kubernetesconfigurationprometheusopen-telemetryexporter

Opentelemetry cant connect to Prometheus on Kubernetes

I have a kubernetes cluster with apps in .net 7 installed . The cluster also has OpenTelemetry as a deployment and Prometheus server installed.

enter image description here

The opentelemetry has a values file as show below

mode: deployment presets:   # enables the k8sattributesprocessor and adds it to the traces, metrics, and logs pipelines   kubernetesAttributes:
    enabled: true   # enables the kubeletstatsreceiver and adds it to the metrics pipelines   kubeletMetrics:
    enabled: true   # Enables the filelogreceiver and adds it to the logs pipelines   logsCollection:
    enabled: true

config:
     exporters:
    otlphttp/metrics:
      endpoint: http://prometheus-server.cmb.svc.cluster.local:9090
      tls:
         insecure: true
    prometheusremotewrite:
      endpoint: http://prometheus-server.cmb.svc.cluster.local:9091/api/v1/write
      tls:
         insecure: true   service:
    pipelines:      
      metrics:                
        exporters:
          - otlphttp/metrics

the Opentelemetry pods starts and runs just fine and is able to receive metrics from all apps on the cluster but it fails to write to the Prometheus cluster. The error generated is given below

2024-03-20T14:54:29.219Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/metrics", "error": "failed to make an HTTP request: Post "http://prometheus-server.cmb.svc.cluster.local:9090/v1/metrics": context deadline exceeded", "interval": "6.716454422s"}

Solution

  • If I'm reading your configuration correctly, you are using the OTLP/HTTP exporter to send the metrics from the OpenTelemetry Collector to Prometheus to the endpoint http://prometheus-server.cmb.svc.cluster.local:9090. Looking at the Prometheus docs, the endpoint should rather be http://prometheus-server.cmb.svc.cluster.local:9090/api/v1/otlp/.

    As described in those docs, you also need to make sure that your Prometheus server is at least at version 2.47 and that the feature flag otlp-write-receiver is enabled.