azurekubernetesingress-controller

502 Bad Gateway and unhealthy probe in AGIC AppGW


My application is with Angular FE and multiple Asp.net core web api's. I have deployed this in AKS and configured Appgw AGIC. My Ingress controller details

Name:             ingress-appgw-deployment-xxxxxxxx-xxxx
Namespace:        kube-system
Priority:         0
Service Account:  ingress-appgw-sa
Node:             aks-agentpool-xxxxxxx-vmss000000/10.224.1.255
Start Time:       Fri, 01 Mar 2024 02:16:47 +0400
Labels:           app=ingress-appgw
                  kubernetes.azure.com/managedby=aks
                  pod-template-hash=6d4f68fc9f
Annotations:      checksum/config: 1e53b6ba909ce787812767da80976912cc41007f8a716c489edab91f215b2bd7
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
                  kubernetes.azure.com/metrics-scrape: true
                  prometheus.io/path: /metrics
                  prometheus.io/port: 8123
                  prometheus.io/scrape: true
                  resource-id:
                    /subscriptions/ea88e1ed-fec0-472f-87f5-a00e6a608f19/resourceGroups/AZ-BELSIO-SOFTWARES-RG/providers/Microsoft.ContainerService/managedClus...
Status:           Running
IP:               10.224.x.xx
IPs:
  IP:           10.224.x.xx
Controlled By:  ReplicaSet/ingress-appgw-deployment-6d4f68fc9f
Containers:
  ingress-appgw-container:
    Container ID:   containerd://de72b0a2d86ffd7e11113b0ee38a374b509b212d56eac166391d28d4ae81849c
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 01 Mar 2024 02:18:47 +0400
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Fri, 01 Mar 2024 02:17:55 +0400
      Finished:     Fri, 01 Mar 2024 02:17:56 +0400
    Ready:          True
    Restart Count:  4
    Limits:
      cpu:     700m
      memory:  600Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-appgw-cm  ConfigMap  Optional: false
    Environment:
      AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
      KUBERNETES_PORT_443_TCP_ADDR:   belsio-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io
      KUBERNETES_PORT:                tcp://myapp-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:        tcp://myapp-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:        myapp-dns-yvu9cnmb.hcp.uaenorth.azmk8s.io
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
      AGIC_POD_NAME:                  ingress-appgw-deployment-6d4f68fc9f-xtlmk (v1:metadata.name)
    Mounts:
      /etc/kubernetes/azure.json from cloud-provider-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9kcms (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloud-provider-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  kube-api-access-9kcms:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

My ingress congiguration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  namespace: default
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  tls:
  - hosts:
    - app.example.com
  rules:
  - host: app.example.com
    http:
        paths:
          - path: /
            pathType: Exact
            backend:
              service:
                name: angular
                port:
                  number: 80
          - path: /api1
            pathType: Prefix
            backend:
              service:
                name: api1
                port:
                  number: 80
          - path: /api2
            pathType: Prefix
            backend:
              service:
                name: api2
                port:
                  number: 80

Expected: domain root path will route to Angular App and /api1 and /api2 to api's. I was able to achieve this in minikube using nginx IC using the same ingress configuration.

Issue: Angular is accessible but cannot access API's. I checked APIs by assigning a public and it's accessible but cannot with the ingress. In the health check, Angular pods are healthy but api pods are unhealth with the below message.

"Received invalid status code: 404 in the backend server’s HTTP response. As per the health probe configuration, 200-399 is the acceptable status code. Either modify probe configuration or resolve backend issues. To learn more visit - https://aka.ms/StatusCodeMismatch."

Please help me fix this and also suggest additional configurations that you think might help me.

Thank you in advance.


Solution

  • The issue was with the routing in the .NET code and was resolved by adding a root prefix for Swagger UI.

    blog by CodeMaze helped. https://code-maze.com/aspnetcore-modify-a-base-uri-of-swagger/