Kibana error: Unable to retrieve version information from Elasticsearch nodes. socket hang up

I am trying to deploy elasticsearch and kibana to kubernetes using this chart and getting this error inside the kibana container, therefore ingress returns 503 error and container is never ready.

Error:

[2022-11-08T12:30:53.321+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. socket hang up - Local: 10.112.130.148:42748, Remote: 10.96.237.95:9200

Ip adress 10.96.237.95 is a valid elasticsearch service address, and port is right.

When i am doing curl to elasticsearch from inside the kibana container, it successfully returns a response.

Am i missing something in my configurations?

Chart version: 7.17.3

Values for elasticsearch chart:

clusterName: "elasticsearch"
nodeGroup: "master"

createCert: false

roles:
  master: "true"
  data: "true"
  ingest: "true"
  ml: "true"
  transform: "true"
  remote_cluster_client: "true"

protocol: https

replicas: 2

sysctlVmMaxMapCount: 262144

readinessProbe:
   failureThreshold: 3
   initialDelaySeconds: 90
   periodSeconds: 10
   successThreshold: 1
   timeoutSeconds: 10

imageTag: "7.17.3"

extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elasticsearch-creds
      key: password
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      name: elasticsearch-creds
      key: username

clusterHealthCheckParams: "wait_for_status=green&timeout=20s"

antiAffinity: "soft"

resources:
  requests:
    cpu: "100m"
    memory: "1Gi"
  limits:
    cpu: "1000m"
    memory: "1Gi"

esJavaOpts: "-Xms512m -Xmx512m"

volumeClaimTemplate:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 30Gi

esConfig:
  elasticsearch.yml: |
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12


secretMounts:
  - name: elastic-certificates
    secretName: elastic-certificates
    path: /usr/share/elasticsearch/config/certs

Values for kibana chart:

elasticSearchHosts: "https://elasticsearch-master:9200"

extraEnvs:
  - name: ELASTICSEARCH_USERNAME
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: username
  - name: ELASTICSEARCH_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elasticsearch-creds
        key: password
  - name: KIBANA_ENCRYPTION_KEY
    valueFrom:
      secretKeyRef:
        name: encryption-key  
        key: encryption_key

kibanaConfig:
  kibana.yml: |
    server.ssl:
      enabled: true
      key: /usr/share/kibana/config/certs/elastic-certificate.pem
      certificate: /usr/share/kibana/config/certs/elastic-certificate.pem
    xpack.security.encryptionKey: ${KIBANA_ENCRYPTION_KEY}
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/elastic-certificate.pem
      verificationMode: certificate
    
protocol: https

secretMounts:
  - name: elastic-certificate-pem
    secretName: elastic-certificate-pem
    path: /usr/share/kibana/config/certs

imageTag: "7.17.3"

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-issuer
    kubernetes.io/ingress.allow-http: 'false'
  paths:
    - path: /
      pathType: Prefix
      backend:
        serviceName: kibana
        servicePort: 5601
  hosts:
    - host: mydomain.com
      paths:
        - path: /
          pathType: Prefix
          backend:
            serviceName: kibana
            servicePort: 5601
  tls:
    - hosts:
        - mydomain.com
      secretName: mydomain.com

UPD: tried it with other image version (8.4.1), nothing has changed, i am getting the same error. By the way, logstash is successfully shipping logs to this elasticsearch instance, so i think problem is in kibana.

Figured it out. It was a complete pain in the ass. I hope these tips will help others:

1. xpack.security.http.ssl.enabled should be set to false. I can't find another way around it, but if you do i'd be glad to hear any advices. As i see it, you don't need security for http layer since kibana connects to elastic via transport layer (correct me if i am wrong). Therefore xpack.security.transport.ssl.enabled should be still set to true, but xpack.security.http.ssl.enabled should be set to false. (don't forget to change your protocol field for readinessProbe to http, and also change protocol for elasticsearch in kibana chart to http.
2. ELASTIC_USERNAME env variable is pointless in elasticsearch chart, only password is used, user is always elastic
3. ELASTICSEARCH_USERNAME in kibana chart should be actually set to kibana_systems user with according password for that user