elasticsearchkibanagoogle-kubernetes-enginekubernetes-ingresselastic-cloud

Exposing Kibana behind GCE ingress (UNHEALTHY state)


I'm trying to expose Kibana behind of a GCE ingress, but the ingress is reporting the kibana service as UNHEALTHY while it is healthy and ready. Just note that the healthcheck created by the Ingress is still using the default value HTTP on the root / and Port: ex:32021. Changing the healthcheck in GCP console to HTTPS on /login and Port: 5601 doesn't change anything and the service is still reported as Unhealthy. The healthcheck port is also being overwritten to the original value, which is strange. I'm using ECK 1.3.1 and below are my configs. I'm I missing anything? Thank you in advance.

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: d3m0
spec:
  version: 7.10.1
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: d3m0
spec:
  version: 7.10.1
  count: 1
  elasticsearchRef:
    name: d3m0
  podTemplate:
    metadata:
      labels:
        kibana: node
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 1Gi
            cpu: 1
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: "/login"
            port: 5601
  http:
    service:
      spec:
        type: NodePort
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
spec:
  backend:
      serviceName: d3m0-kb-http
      servicePort: 5601

Solution

  • When using ECK, all the security feature are enabled on ES and Kibana, which means that their services do not accept HTTP traffic used by the default GCP loadbalancer Healthcheck. You must add the required annotations to the services and override the healthcheck paths as in the code below. Please find more details here.

        apiVersion: kibana.k8s.elastic.co/v1
        kind: Kibana
        metadata:
          name: d3m0
        spec:
          version: 7.10.1
          count: 1
          elasticsearchRef:
            name: d3m0
          http:
            service:
              metadata:
                labels:
                  app: kibana
                annotations:
                  # Enable TLS between GCLB and the application
                  cloud.google.com/app-protocols: '{"https":"HTTPS"}'
                  service.alpha.kubernetes.io/app-protocols: '{"https":"HTTPS"}'
                  # Uncomment the following line to enable container-native load balancing.
                  cloud.google.com/neg: '{"ingress": true}'
        
          podTemplate:
            metadata:
              labels:
                name: kibana-fleet
            spec:
              containers:
              - name: kibana
                resources:
                  limits:
                    memory: 1Gi
                    cpu: 1
                readinessProbe:
                      # Override the readiness probe as GCLB reuses it for its own healthchecks
                      httpGet:
                        scheme: HTTPS
                        path: "/login"
                        port: 5601