sslkubernetes-helmgrafanakubernetes-ingresskube-prometheus-stack

Securing grafana ingress with tls in kube-prometheus-stack values.yaml and make grafana available via https


I am using kube-prometheus-stack to monitor my system in gcp. Due to new requirements all my ingress need to be secured with tls. As a first step I wanted to make the grafana webpage available via https.

I created a tls secret and updated my values.yaml. After helm upgrade everything seems to work fine but page is still available via http only.

Hope you can support me here.

    grafana:
  enabled: true
  namespaceOverride: ""

  ## Deploy default dashboards.
  ##
  defaultDashboardsEnabled: true

  adminPassword: prom-operator

  ingress:
    ## If true, Grafana Ingress will be created
    ##
    enabled: true

    ## Annotations for Grafana Ingress
    ##
#    annotations: {
#      kubernetes.io/ingress.class: gce-internal
      # kubernetes.io/tls-acme: "true"
#    }
    ## Labels to be added to the Ingress
    ##
    labels: {}

    ## Hostnames.
    ## Must be provided if Ingress is enable.
    ##
    # hosts:
    #   - grafana.domain.com
    hosts: []

    ## Path for grafana ingress
    # path: /*

    ## TLS configuration for grafana Ingress
    ## Secret must be manually created in the namespace
    ##
    tls:
      - secretName: monitoring-tls-secret
    #   hosts:
    #     - grafana.example.com

in the meantime I decided to create the ingress a different way.

I created a ssl-certificate and try to use that instead.

When starting up I get the failure down below. Which is strange as kubernetes.io/ingress.allow-http is configured.

 kubectl describe ingress monitoring-cl2-grafana -n monitoring-cl2
Name:             monitoring-cl2-grafana
Namespace:        monitoring-cl2
Address:          x.x.x.x
Default backend:  default-http-backend:80 (y.y.y.y:8080)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /*   monitoring-cl2-grafana:80 (<deleted>)
Annotations:  ingress.gcp.kubernetes.io/pre-shared-cert: monitoring-ssl
              ingress.kubernetes.io/backends:
                {"k8s1-613c3440-kube-system-default-http-backend-80-240d1018":"HEALTHY","k8s1-613c3440-mtx-monitoring--mtx-monitoring-cl2-gra-8-f146f2b2":...
              ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-3s1rnwzg-monitoring--monitoring-cl2-gr-hgx28ojy
              ingress.kubernetes.io/https-target-proxy: k8s2-ts-3s1rnwzg-monitoring--monitoring-cl2-gr-hgx28ojy
              ingress.kubernetes.io/ssl-cert: monitoring-ssl
              ingress.kubernetes.io/url-map: k8s2-um-3s1rnwzg-monitoring--monitoring-cl2-gr-hgx28ojy
              kubernetes.io/ingress.allow-http: false
              kubernetes.io/ingress.class: gce-internal
              kubernetes.io/ingress.global-static-ip-name: grafana-cl2
              meta.helm.sh/release-name: monitoring-cl2
              meta.helm.sh/release-namespace: monitoring-cl2
Events:
  Type     Reason                    Age                  From                     Message
  ----     ------                    ----                 ----                     -------
  Warning  Sync                      34m (x12 over 35m)   loadbalancer-controller  Error syncing to GCP: error running load balancer syncing routine: loadbalancer 3s1rnwzg-mtx-monitoring--monitoring-cl2-gr-hgx28ojy does not exist: error invalid internal ingress https config
  Warning  WillNotConfigureFrontend  26m (x18 over 35m)   loadbalancer-controller  gce-internal Ingress class does not currently support both HTTP and HTTPS served on the same IP (kubernetes.io/ingress.allow-http must be false when using HTTPS).
  Normal   Sync                      3m34s                loadbalancer-controller  TargetProxy "k8s2-ts-3s1rnwzg-monitoring--monitoring-cl2-gr-hgx28ojy" certs updated
  Normal   Sync                      3m29s (x9 over 35m)  loadbalancer-controller  Scheduled for sync

grafana:
   image:
     repository: europe-west3-docker.pkg.dev/<deleted info>/grafana
     tag: 7.5.5
     sha: ""
   sidecar:
     image:
       repository: europe-west3-docker.pkg.dev/<deleted info>/prometheus/k8s-sidecar
       tag: 1.10.7
       sha: ""
     imagePullPolicy: IfNotPresent
   service:
     enabled: true
     type: NodePort

     annotations: {
       cloud.google.com/neg: '{"ingress": true}'
     }
     labels: {}
     portName: service
   ingress:
     enabled: true
     path: /*
     pathType: ImplementationSpecific
     annotations: {
       ingress.gcp.kubernetes.io/pre-shared-cert: "monitoring-ssl",
       kubernetes.io/ingress.allow-http: "false",
       kubernetes.io/ingress.class: "gce-internal",
       kubernetes.io/ingress.global-static-ip-name: "grafana-cl2"
     }


Solution

  • WORKING NOW WITH FOLLOWING CONFIG

    grafana:
       image:
         repository: europe-west3-docker.pkg.dev/del/mtx-monitoring/prometheus/grafana
         tag: 7.5.5
         sha: ""
       sidecar:
         image:
           repository: europe-west3-docker.pkg.dev/del/mtx-monitoring/prometheus/k8s-sidecar
           tag: 1.10.7
           sha: ""
         imagePullPolicy: IfNotPresent
       service:
         enabled: true
         type: NodePort
    #     port: 80
     #    targetPort: 3000
         annotations: {
           cloud.google.com/neg: '{"ingress": true}'
         }
         labels: {}
         portName: service
       ingress:
         enabled: true
         path: /*
         pathType: ImplementationSpecific
         annotations: {
           ingress.gcp.kubernetes.io/pre-shared-cert: "monitoring-ssl",
           kubernetes.io/ingress.allow-http: "false",
           kubernetes.io/ingress.class: "gce-internal",
           kubernetes.io/ingress.global-static-ip-name: "grafana-cl2"
         }
       spec:
         rules:
         - host: grafana.monitoring.com
         http:
          paths:
          - backend:
              service:
                name: mtx-monitoring-cl2-grafana
                port:
                  number: 80