In a hierarchical federated setup of prometheus with a Pull model for the metrics, I see "prometheus" and "prometheus_replica" labels in the metrics that's captured. The system is monitoring a StatefulSet deployment of Kubernetes.
When querying or alerting I see duplicate data included due to these labels, i.e I see a metric with these labels and also without these. Effectively causing wrong counts and alerts.
I see "prometheus" and "prometheus_replica" labels used in the queries on the prometheus that pulls metrics from federated endpoint.
I use ServiceMonitor with Prometheus operator on every kube cluster. All the metrics is federated to a single different Prometheus where this problem is seen.
Is there any documentation on how these labels get generated? Are those metrics to be treated duplicate or ignored?
I finally found these labels coming from the prometheus operator. It was added for an requirement that's unwritten in any documents. I see it doesn't work in 0.17 version. Its works in 0.23 version of operator.