APISIX ingress controller on Kind cluster not routing requests correctly

I am trying to setup APISIX gateway ingress controller enabled application (spring-boot) on a local KIND kubernetes cluster. Here are the steps that I followed,

I was able to succesfully install and configure the APISIX gateway (PORT: 8090) in my local kind kubernetes cluster. I can confirm that an ingress manifest is translated to an APISIX route correctly. The target application is configured with a service "client-app" and I can confirm that it does have the corresponding endpoints configured correctly.

Setup

Ingress YAML:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpserver-ingress
  namespace: oidcapp
spec:
  # we use APISIX Ingress and it watches Ingress resources with "apisix" ingressClassName
  ingressClassName: apisix
  rules:
  - host: authclient.com
    http:
      paths:
      - backend:
          service:
            name: client-app
            port:
              number: 80
        path: /oidcapp
        pathType: Prefix

Target Application service:

$ kubectl describe svc -n oidcapp client-app 
Name:                     client-app
Namespace:                oidcapp
Labels:                   app=client-app
Annotations:              <none>
Selector:                 app=client-app
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.96.136.11
IPs:                      10.96.136.11
Port:                     http  80/TCP
TargetPort:               8080/TCP
Endpoints:                10.244.1.23:8080
Session Affinity:         None
Internal Traffic Policy:  Cluster

APISIX route mapping:

{
    "createdIndex": 11332,
    "key": "/apisix/routes/33d660f9",
    "modifiedIndex": 11532,
    "value": {
        "priority": 0,
        "status": 1,
        "uris": [
            "/oidcapp",
            "/oidcapp/*"
        ],
        "name": "ing_oidcapp_httpserver-ingress_4cafc3f3",
        "id": "33d660f9",
        "upstream_id": "fdcb23fc",
        "host": "authclient.com",
        "create_time": 1744820603,
        "update_time": 1744851162,
        "desc": "Created by apisix-ingress-controller, DO NOT modify it manually",
        "labels": {
            "managed-by": "apisix-ingress-controller"
        }
    }
}

I then installed a Loadbalancer in Kind cluster, following the instructions below: Kind Docs | LoadBalancer

Based on these instructions, added the following Service manifest:

kind: Service
apiVersion: v1
metadata:
  name: apisix-gateway-service
  namespace: apisix
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: apisix
  ports:
  - port: 5678
    targetPort: 8090

In my etc/hosts, I have configured the following host-mapping:

172.18.0.2      authclient.com

Here are the relevant kubectl command output:

$ kubectl get svc -n apisix

NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
apisix-admin             ClusterIP      10.96.220.99    <none>        9180/TCP            7d18h
apisix-gateway-service   LoadBalancer   10.96.183.128   172.18.0.2    5678:31448/TCP      80m
etcd-headless            ClusterIP      None            <none>        2379/TCP,2380/TCP   7d18h

Problem

When I execute the following curl command, I get a connection reset error:

$ curl -v http://authclient.com:5678/oidcapp
*   Trying 172.18.0.2:5678...
* TCP_NODELAY set
* Connected to authclient.com (172.18.0.2) port 5678 (#0)
> GET /oidcapp HTTP/1.1
> Host: authclient.com:5678
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

I expect the CURL request to be forwarded to an apisix-gateway pod and then routed to one of the service endpoints determined by the gateway's route mapping.

cURL ---> Kind LoadBalancer ---> APISIX ---> client-app

Unfortunately, the logs of load balancer and apisix-gateway-ingress-controller does not provide any further details.

Solution

TL;DR

The problem is that you are binding the Service to the wrong port: 8090.

Explaination

The reason why your CURL request fails, is that:

in APISIX container, port 8090 is not mapped to anything;
even if it were, there's nothing that is listening on that port.

In fact, by default, APISIX listens for HTTP traffic on port 9080. Therefore, the YAML for your Service should look like this:

kind: Service
apiVersion: v1
metadata:
  name: apisix-gateway-service
  namespace: apisix
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: apisix
  ports:
  - port: 5678
    targetPort: 9080 # <-- APISIX proxy port

Some useful references: