kubernetesprometheusprometheus-operatorkube-prometheus-stackservicemonitor

Using namespaceSelector in ServiceMonitor Kube-prometheus


I'm studying Prometheus Operator from the project: https://github.com/prometheus-operator/kube-prometheus

When my service/deployment is in the same "monitoring" namespace as prometheus I have success to visualize metrics. Example:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: caixa-app
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-caixa
    app.kubernetes.io/part-of: kube-prometheus
spec:
  endpoints:
    - interval: 15s
      port: http
  selector:
    matchLabels:
      app: caixa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: caixa-deployment
  namespace: monitoring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: caixa-pod
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "80"
      labels:
        app: caixa-pod
    spec:
      containers:
      - name: caixa
        image: 'REGISTRY-NAME/caixa:1.0'
        ports:
          - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
      imagePullSecrets:
      - name: regcred      
---
apiVersion: v1
kind: Service
metadata:
  name: caixa-svc
  namespace: monitoring
  labels:
    app: caixa
spec:
  selector:
    app: caixa-pod
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  type: ClusterIP

In the way mentioned above I have success in the same namespace "monitoring"

targets

However I need my Prometheus-operator to collect metrics from other namespaces that I will create. So I used the parameter "namespaceSelector".

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: caixa-app
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-caixa
    app.kubernetes.io/part-of: kube-prometheus
spec:
  endpoints:
    - interval: 15s
      port: http
  namespaceSelector:
    matchNames:
    - shop
  selector:
    matchLabels:
      app: caixa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: caixa-deployment
  namespace: shop
spec:
  replicas: 2
  selector:
    matchLabels:
      app: caixa-pod
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: /metrics
        prometheus.io/port: "80"
      labels:
        app: caixa-pod
    spec:
      containers:
      - name: caixa
        image: 'REGISTRY-NAME/caixa:1.0'
        ports:
          - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
      imagePullSecrets:
      - name: regcred      
---
apiVersion: v1
kind: Service
metadata:
  name: caixa-svc
  namespace: shop
  labels:
    app: caixa
spec:
  selector:
    app: caixa-pod
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
  type: ClusterIP

That way the metrics don't appear in prometheus, but they are in my Prometheus config.

target2

- job_name: serviceMonitor/monitoring/caixa-app/0
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
....
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - shop

Do I need to add any more elements? so that Service Monitor can collect metrics from other namespaces?

I already validated in my ClusterRole and Prometheus Operator has all permissions in namespaces.


Solution

  • Problem solved!

    In Kube-prometheus by default it maps namespaces: (default, kube-system, monitoring)

    To add specific namespaces we must edit the two files: prometheus-roleSpecificNamespaces.yaml

    - apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        labels:
          app.kubernetes.io/component: prometheus
          app.kubernetes.io/instance: k8s
          app.kubernetes.io/name: prometheus
          app.kubernetes.io/part-of: kube-prometheus
          app.kubernetes.io/version: 2.38.0
        name: prometheus-k8s
        namespace: loja
      rules:
      - apiGroups:
        - ""
        resources:
        - services
        - endpoints
        - pods
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - extensions
        resources:
        - ingresses
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - networking.k8s.io
        resources:
        - ingresses
        verbs:
        - get
        - list
        - watch- apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        labels:
          app.kubernetes.io/component: prometheus
          app.kubernetes.io/instance: k8s
          app.kubernetes.io/name: prometheus
          app.kubernetes.io/part-of: kube-prometheus
          app.kubernetes.io/version: 2.38.0
        name: prometheus-k8s
        namespace: loja
      rules:
      - apiGroups:
        - ""
        resources:
        - services
        - endpoints
        - pods
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - extensions
        resources:
        - ingresses
        verbs:
        - get
        - list
        - watch
      - apiGroups:
        - networking.k8s.io
        resources:
        - ingresses
        verbs:
        - get
        - list
        - watch
    

    prometheus-roleBindingSpecificNamespaces.yaml

    - apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        labels:
          app.kubernetes.io/component: prometheus
          app.kubernetes.io/instance: k8s
          app.kubernetes.io/name: prometheus
          app.kubernetes.io/part-of: kube-prometheus
          app.kubernetes.io/version: 2.38.0
        name: prometheus-k8s
        namespace: loja
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: prometheus-k8s
      subjects:
      - kind: ServiceAccount
        name: prometheus-k8s
        namespace: monitoring
    

    After adding the desired namespace and applying the yml, the ServiceMonitor will be added.