502 bad gateway error of ALB & Istio when using https

I got 502 bad gateway error for https when using Istio and AWS ALB.

For some reason, I have to use ALB ingress before my Istio ingressgateway, and I also need to use https to connect from my ingress to istio ingressgateway. But I got the 502 bad gateway error. If I use http, it works fine.

I can find the following information in the logs of istio ingressgateway: "response_code_details": "filter_chain_not_found"

Does someone have any idea?

The following is my Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-gateway
  namespace: istio-system
  annotations:
    alb.ingress.kubernetes.io/group.name: <group name>
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: <my arn>
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: <my subnet>
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
spec:
  ingressClassName: alb
  rules:
  - host: "my.hostname.com"
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 443
  tls:
  - hosts:
    - "my.hostname.com"

The following is my istio-ingressgateway

...
          serviceAnnotations:
            alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
            alb.ingress.kubernetes.io/healthcheck-port: "30218"
          service:
            type: NodePort
            ports:
            - name: http
              port: 80
              protocol: TCP
              targetPort: 8080
            - name: https
              port: 443
              protocol: TCP
              targetPort: 8443
            - name: status-port
              nodePort: 30218
              port: 15021
              protocol: TCP
              targetPort: 15021
...

The following is my Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - my.hostname.com
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - my.hostname.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: my-tls-cert
      mode: SIMPLE
      privateKey: sds
      serverCertificate: sds

It works fine if I change the ingress to use http as following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-gateway
  namespace: istio-system
  annotations:
    alb.ingress.kubernetes.io/group.name: <group name>
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: <my arn>
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/subnets: <my subnet>
spec:
  ingressClassName: alb
  rules:
  - host: "my.hostname.com"
    http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80
  tls:
  - hosts:
    - "my.hostname.com"

ALB does not include the SNI extension when performing TLS handshake with your Istio Ingress Gateway. SNI is needed for route matching when you include 'host' in your Gateway resource. This is documented here:

https://istio.io/latest/docs/ops/common-problems/network-issues/#configuring-sni-routing-when-not-sending-sni

The workaround is to configure the Istio 443 listener with hosts: '*', to avoid SNI matching, and then specify the host in the VirtualService resource for routing. You may also do some header matching in the VirtualService resource.

AWS have stated that including the SNI from client --> ALB connection is on their list, but their is no roadmap or ETA.

Also check these:

https://repost.aws/questions/QUOg0LUwafRFaorbsrYDP7xA/does-alb-send-sni-information-in-tls-handshake-to-a-back-end-server

https://kevin.burke.dev/kevin/amazons-albs-insecure-internal-traffic/

Updated Gateway configuration from question

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - my.hostname.com
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - "*"
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: my-tls-cert
      mode: SIMPLE
      privateKey: sds
      serverCertificate: sds