I would like to use otel-collector to scrap multiple business pods.
This solution is already working for one (but just one) pod:
kubectl get pods
mycoolbusinesspod-7b4f8f4c4f-74sqq 1/1 Running 0
The concept is that there is one business pod which generates the metrics.
I expose a Kubernetes service svc for this business pod.
The collector is configured to scrap this only one endpoint and path.
---
apiVersion: v1
kind: ConfigMap
metadata:
name: otel-collector-conf
labels:
app: opentelemetry
component: otel-collector-conf
data:
otel-collector-config: |
receivers:
prometheus:
config:
scrape_configs:
- job_name: "jobname"
scrape_interval: 5s
metrics_path: '/actuator/prometheus'
static_configs:
- targets: ["mycoolbusinesspod-svc:8080"]
processors:
batch:
exporters:
prometheus:
endpoint: "localhost:8889"
service:
pipelines:
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [prometheus]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: otel-collector
labels:
app: opentelemetry
component: otel-collector
spec:
selector:
matchLabels:
app: opentelemetry
component: otel-collector
minReadySeconds: 5
progressDeadlineSeconds: 120
replicas: 1
template:
metadata:
labels:
app: opentelemetry
component: otel-collector
spec:
containers:
- command:
- "/otelcol"
- "--config=/conf/otel-collector-config.yaml"
image: otel/opentelemetry-collector:latest
name: otel-collector
resources:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 200m
memory: 400Mi
ports:
- containerPort: 55679 # Default endpoint for ZPages.
- containerPort: 4317 # Default endpoint for OpenTelemetry receiver.
- containerPort: 14250 # Default endpoint for Jaeger gRPC receiver.
- containerPort: 14268 # Default endpoint for Jaeger HTTP receiver.
- containerPort: 9411 # Default endpoint for Zipkin receiver.
- containerPort: 8888 # Default endpoint for querying metrics.
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: GOMEMLIMIT
value: 1600MiB
volumeMounts:
- name: otel-collector-config-vol
mountPath: /conf
This is working, I can see the metrics.
However, now, I have multiple replicas.
kubectl get pods
mycoolbusinesspod-7b4f8f4c4f-lrphj 1/1 Running 0
mycoolbusinesspod-7b4f8f4c4f-n7v9h 1/1 Running 0
mycoolbusinesspod-7b4f8f4c4f-jght9 1/1 Running 0
Attempt 1:
I see the configuration accepts an array for "target". But I do not know what to input in the array.
Attempt 2:
I do some kind of load balancer as service to load balance between the multiple pods.
However, this will not get the metrics from the pods that were not load balanced.
How to configure Otel-collector to scrap metrics for all the business pods?
I do some kind of load balancer as service to load balance between the multiple pods. However, this will not get the metrics from the pods that were not load balanced.
Since you're looking to scrape configuration on every pods, I think setting pod's annotation and use Prometheus kubernetes service discovery to scrape pods metrics directly should be the way to go in microservices deployment.
If you'd scrape through k8s ClusterIP service the scrape request will get randomly distributed to every pods you deploy, which is undesirable since we want to know metrics of every pods.
I believe otel collector support this feature out of the box. ref
Sample configuration could be like this:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
this configuration will make otel collector scrape metrics from any pods with annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/metrics"
prometheus.io/port: "8080"
while keeping kubernetes_pod_name
and kubernetes_namespace
value as metadata in labels.
You might need to set appropriate ClusterRole for your otel-collector ServiceAccount though.
something like:
rules:
- apiGroups: [""]
resources:
- pods
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
should suffice.