kong: gke ingress: An invalid response was received from the upstream server

I have kong ingress on GKE as follows, which has a default path "/" forwards to a web application written in nextjs that has keycloak integration.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myprojqlfingress
  namespace: qlf
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: letsencrypt-qlf
    kubernetes.io/ingress.class: kong
    konghq.com/request-buffering: "true"
    konghq.com/response-buffering: "true"
spec:
  tls:
  - secretName: myproject-qlf
    hosts:
    - myproject-qlf.mydomain.net
  rules:
  - host: myproject-qlf.mydomain.net
    http:
      paths:
      - path: /rfm
        pathType: ImplementationSpecific
        backend:
          service:
            name: rfmapi
            port:
              number: 8091
      - path: /mb
        pathType: ImplementationSpecific
        backend:
          service:
            name: mbapi
            port:
              number: 8094
      - path: /custseg
        pathType: ImplementationSpecific
        backend:
          service:
            name: custapi
            port:
              number: 8093
      - path: /energy
        pathType: ImplementationSpecific
        backend:
          service:
            name: energyapi
            port:
              number: 8097
      - path: /airline
        pathType: ImplementationSpecific
        backend:
          service:
            name: airlineapi
            port:
              number: 8096
      - path: /hotel
        pathType: ImplementationSpecific
        backend:
          service:
            name: hotelapi
            port:
              number: 8095
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: datainsightfrontwebapp
            port:
              number: 3000

Here I have changed domain name for confidentiality reason.

So after authentication it is throwing error:

An invalid response was received from the upstream server

The same problem was there in dev environment also which I fixed it using the annotation: konghq.com/request-buffering: "true" but in this case it is not working.

certificate and all configurations are working fine.

Any idea?

I fixed the issue using following manifest file:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ingress-kong
  name: ingress-kong
  namespace: kong
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-kong
  template:
    metadata:
      annotations:
        kuma.io/gateway: enabled
        traffic.sidecar.istio.io/includeInboundPorts: ""
      labels:
        app: ingress-kong
    spec:
      containers:
      - env:
        - name: KONG_PROXY_LISTEN
          value: 0.0.0.0:8000, 0.0.0.0:8443 ssl http2
        - name: KONG_PORT_MAPS
          value: 80:8000, 443:8443
        - name: KONG_ADMIN_LISTEN
          value: 127.0.0.1:8444 ssl
        - name: KONG_STATUS_LISTEN
          value: 0.0.0.0:8100
        - name: KONG_DATABASE
          value: "off"
        - name: KONG_NGINX_WORKER_PROCESSES
          value: "2"
        - name: KONG_KIC
          value: "on"
        - name: KONG_ADMIN_ACCESS_LOG
          value: /dev/stdout
        - name: KONG_ADMIN_ERROR_LOG
          value: /dev/stderr
        - name: KONG_PROXY_ERROR_LOG
          value: /dev/stderr
        - name: KONG_CLIENT_BODY_BUFFER_SIZE
          value: 124m
        - name: KONG_CLIENT_MAX_BODY_SIZE
          value: "0"
        - name: KONG_NGINX_PROXY_PROXY_BUFFER_SIZE
          value: 160k
        - name: KONG_NGINX_PROXY_PROXY_BUFFERS
          value: 64 160k
        image: kong:2.7
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - kong quit
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 8100
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: proxy
        ports:
        - containerPort: 8000
          name: proxy
          protocol: TCP
        - containerPort: 8443
          name: proxy-ssl
          protocol: TCP
        - containerPort: 8100
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /status
            port: 8100
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
      - env:
        - name: CONTROLLER_KONG_ADMIN_URL
          value: https://127.0.0.1:8444
        - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
          value: "true"
        - name: CONTROLLER_PUBLISH_SERVICE
          value: kong/kong-proxy
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: kong/kubernetes-ingress-controller:2.2.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: ingress-controller
        ports:
        - containerPort: 8080
          name: webhook
          protocol: TCP
        - containerPort: 10255
          name: cmetrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
      serviceAccountName: kong-serviceaccount

Which means below parameters were essentials:

- name: KONG_CLIENT_BODY_BUFFER_SIZE
  value: 124m
- name: KONG_CLIENT_MAX_BODY_SIZE
  value: "0"
- name: KONG_NGINX_PROXY_PROXY_BUFFER_SIZE
  value: 160k
- name: KONG_NGINX_PROXY_PROXY_BUFFERS
  value: 64 160k