nginxprometheusmicrok8s

Nginx metrics not appearing on prometheus


I have a single node kubernetes cluster deployed by microk8s. I'm using the default Nginx ingress controller and the prometheus addon. I'm trying to collect the nginx metrics on prometheus so that I monitor it later on grafana. The problem is that I'm not getting the metrics on Prometheus after creating the serviceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-prom-stack-nginx-ingress-microk8s
  namespace: observability
  annotations:
    meta.helm.sh/release-name: kube-prom-stack
    meta.helm.sh/release-namespace: observability
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: kube-prom-stack
    app.kubernetes.io/name: nginx-ingress-microk8s
    app.kubernetes.io/part-of: nginx-ingress-microk8s
    release: kube-prom-stack
spec:
  selector:
    matchLabels:
      microk8s-application: nginx-ingress-microk8s
  endpoints:
    - honorLabels: true
      enableHttp2: true
      port: health
      path: /metrics
      interval: 5s
      scheme: http
  namespaceSelector:
    matchNames:
      - ingress
      - observability
      - kube-system
      - default

Prometheus seems to detect the scrape pool but can't find any endpoint associated with it.

prometheus targets

Nginx seems to be exporting the metrics correctly. Nginx metrics

The serviceMonitor seems to have the following prometheus job:

- job_name: serviceMonitor/observability/kube-prom-stack-nginx-ingress-microk8s/0
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_microk8s_application, __meta_kubernetes_service_labelpresent_microk8s_application]
    separator: ;
    regex: (nginx-ingress-microk8s);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: health
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_container_name]
    separator: ;
    regex: (.*)
    target_label: container
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_phase]
    separator: ;
    regex: (Failed|Succeeded)
    replacement: $1
    action: drop
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: health
    action: replace
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    modulus: 1
    target_label: __tmp_hash
    replacement: $1
    action: hashmod
  - source_labels: [__tmp_hash]
    separator: ;
    regex: "0"
    replacement: $1
    action: keep
  kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
      - ingress
      - observability
      - kube-system
      - default

Solution

  • Finally I found out that when deploying the Nginx ingress controller, microk8s doesn't create a service associated to it. I had to do it manually so that it could be discovered by the prometheus ServiceMonitor. Here is the created service:

    apiVersion: v1
    kind: Service
    metadata:
      name: kube-prom-stack-nginx-ingress-microk8s
      namespace: ingress
      labels:
        microk8s-application: nginx-ingress-microk8s
        release: kube-prom-stack
    spec:
      ports:
        - port: 10254
          targetPort: 10254
          name: prometheus
          protocol: TCP
      selector:
        microk8s-application: nginx-ingress-microk8s
    

    enter image description here