otel-collector to scrap multiple pods

I would like to use otel-collector to scrap multiple business pods.

This solution is already working for one (but just one) pod:

kubectl get pods

mycoolbusinesspod-7b4f8f4c4f-74sqq 1/1 Running 0

The concept is that there is one business pod which generates the metrics.

I expose a Kubernetes service svc for this business pod.

The collector is configured to scrap this only one endpoint and path.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: "jobname"
              scrape_interval: 5s
              metrics_path: '/actuator/prometheus'
              static_configs:
                - targets: ["mycoolbusinesspod-svc:8080"]
    
    processors:
      batch:
    
    exporters:
      prometheus:
        endpoint: "localhost:8889"
    
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [batch]
          exporters: [prometheus]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-collector
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-collector
    spec:
      containers:
        - command:
            - "/otelcol"
            - "--config=/conf/otel-collector-config.yaml"
          image: otel/opentelemetry-collector:latest
          name: otel-collector
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi
          ports:
            - containerPort: 55679 # Default endpoint for ZPages.
            - containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
            - containerPort: 14250 # Default endpoint for Jaeger gRPC receiver.
            - containerPort: 14268 # Default endpoint for Jaeger HTTP receiver.
            - containerPort: 9411 # Default endpoint for Zipkin receiver.
            - containerPort: 8888  # Default endpoint for querying metrics.
          env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: GOMEMLIMIT
              value: 1600MiB
          volumeMounts:
            - name: otel-collector-config-vol
              mountPath: /conf

This is working, I can see the metrics.

However, now, I have multiple replicas.

kubectl get pods

mycoolbusinesspod-7b4f8f4c4f-lrphj 1/1 Running 0
mycoolbusinesspod-7b4f8f4c4f-n7v9h 1/1 Running 0
mycoolbusinesspod-7b4f8f4c4f-jght9 1/1 Running 0

Attempt 1:

I see the configuration accepts an array for "target". But I do not know what to input in the array.

Attempt 2:

I do some kind of load balancer as service to load balance between the multiple pods.

However, this will not get the metrics from the pods that were not load balanced.

How to configure Otel-collector to scrap metrics for all the business pods?

I do some kind of load balancer as service to load balance between the multiple pods. However, this will not get the metrics from the pods that were not load balanced.

Since you're looking to scrape configuration on every pods, I think setting pod's annotation and use Prometheus kubernetes service discovery to scrape pods metrics directly should be the way to go in microservices deployment.
If you'd scrape through k8s ClusterIP service the scrape request will get randomly distributed to every pods you deploy, which is undesirable since we want to know metrics of every pods.

I believe otel collector support this feature out of the box. ref

Sample configuration could be like this:

      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

this configuration will make otel collector scrape metrics from any pods with annotations:

  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "8080"

while keeping kubernetes_pod_name and kubernetes_namespace value as metadata in labels.

---

You might need to set appropriate ClusterRole for your otel-collector ServiceAccount though.
something like:

rules:
- apiGroups: [""]
  resources:
  - pods
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

should suffice.