How can we include custom labels/annotations of K8s objects in Prometheus metrics?

For example I have StatefulSet with custom labels/annotations applied on it.

Everyone mentions these two metrics should be used, because kube-state-metrics should generate labels/annotations as well.

kube_statefulset_annotations
kube_statefulset_labels

The thing is, I can see only default ones (job,instance, namespace,...) but not additionally added labels/annotations.

Example of the manifest I am testing with:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    label_network: "111"
  labels:
    app: testing-labels
    label_STATEFULSET_LABEL: "111"
    label_network: "111"
  name: testing-labels
  namespace: sre-test
spec:
  selector:
    matchLabels:
      app: testing-labels
  serviceName: testing-labels-headless
  template:
    metadata:
      labels:
        app: testing-labels
        label_network: "111"

I've added so many different labels/annotations but the kube_statefulset_labels{statefulset='testing-labels'} returns:

kube_statefulset_labels{container="kube-rbac-proxy-main", instance="10.2.23.229:8443", job="kube-state-metrics", namespace="sre-test", prometheus="aws-monitoring/k8s", prometheus_replica="prometheus-k8s-1", statefulset="testing-labels"}

...which certainly doesn't contain any extra label. Any idea would be helpful? 🙏🏼

Versions:

• kube-state-metrics:v2.5.0
• kube-rbac-proxy:v0.12.0

In order to get custom metrics within kube_statefulset_labels we need to add

--metric-labels-allowlist as a flag in kube-state-metrics.

In this particular example it would be:

--metric-labels-allowlist=statefulsets=[label_network]