I've been looking through a lot of posts of 503 errors when using ingress-nginx but haven't come across a solution that works with my setup.
Running on a local Oracle Linux Server v8.5. Using k3s, and the following info for the node:
kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
mptdevws00 Ready master 6d21h v1.19.2+k3s1 192.168.10.100 <none> Oracle Linux Server 8.5 5.4.17-2102.201.3.el8uek.x86_64 containerd://1.4.0-k3s1
Running the bare-metal ingress-nginx located at https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/baremetal/deploy.yaml
I'm deploying a very basic hello-word deployment, service and ingress, defined in the following:
---
apiVersion: v1
kind: Service
metadata:
name: hello-world-service
spec:
type: NodePort
ports:
- port: 80
selector:
app: hello-world
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world-test
spec:
selector:
matchLabels:
app: hello-world
replicas: 1
template:
metadata:
labels:
app: hello-world
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
volumeMounts:
- name: hello-world-volume
mountPath: /usr/share/nginx/html
volumes:
- name: hello-world-volume
configMap:
name: hello-world
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: hello-world-ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:
- http:
paths:
- path: /hello
pathType: Prefix
backend:
service:
name: hello-world-service
port:
number: 80
Had to run mod_probe ip_tables to get that loaded, and I have to run kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission each time I launch the ingress-nginx deployment because the validation webhook continually spits out an Unknown Host error, so I just turned that piece off until I get this working.
Pods, services, ingress, and endpoints seem to launch fine:
kubectl get pods --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system metrics-server-7b4f8b595-kp4j4 1/1 Running 0 6d21h 10.42.0.2 mptdevws00 <none> <none>
kube-system local-path-provisioner-7ff9579c6-s6f4z 1/1 Running 1 6d21h 10.42.0.4 mptdevws00 <none> <none>
kube-system coredns-66c464876b-4g5q6 1/1 Running 0 6d21h 10.42.0.3 mptdevws00 <none> <none>
ingress-nginx svclb-ingress-nginx-controller-hc8gb 2/2 Running 0 58m 10.42.0.206 mptdevws00 <none> <none>
ingress-nginx ingress-nginx-admission-create-m9nfw 0/1 Completed 0 58m 10.42.0.207 mptdevws00 <none> <none>
ingress-nginx ingress-nginx-admission-patch-kdvdf 0/1 Completed 0 58m 10.42.0.208 mptdevws00 <none> <none>
ingress-nginx ingress-nginx-controller-5979dd67b-lrg5r 1/1 Running 0 58m 10.42.0.209 mptdevws00 <none> <none>
default hello-world-test-59cc4bc76f-b58k9 1/1 Running 0 57m 10.42.0.212 mptdevws00 <none> <none>
kubectl get service --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 6d21h <none>
kube-system kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP 6d21h k8s-app=kube-dns
kube-system metrics-server ClusterIP 10.43.191.252 <none> 443/TCP 6d21h k8s-app=metrics-server
ingress-nginx ingress-nginx-controller-admission ClusterIP 10.43.141.95 <none> 443/TCP 59m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
ingress-nginx ingress-nginx-controller LoadBalancer 10.43.191.206 192.168.10.100 80:32700/TCP,443:31849/TCP 59m app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
default hello-world-service NodePort 10.43.56.86 <none> 80:31043/TCP 58m app=hello-world
kubectl get ingress --all-namespaces -o wide
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
default hello-world-ingress nginx * 192.168.10.100 80 49m
kubectl get ep --all-namespaces -o wide
NAMESPACE NAME ENDPOINTS AGE
default kubernetes 192.168.10.100:6443 6d21h
kube-system metrics-server 10.42.0.2:443 6d21h
kube-system kube-dns 10.42.0.3:53,10.42.0.3:9153,10.42.0.3:53 6d21h
ingress-nginx ingress-nginx-controller 10.42.0.209:443,10.42.0.209:80 60m
ingress-nginx ingress-nginx-controller-admission 10.42.0.209:8443 60m
default hello-world-service 10.42.0.212:80 59m
kube-system rancher.io-local-path <none> 6d21h
If I do a kubectl exec -it --namespace=ingress-nginx ingress-nginx-controller-5979dd67b-lrg5r -- curl http://hello-world-service.default.svc/ it returns the basic Hello World html fine, so I know the service and deployment are at least working correctly.
If I do curl http://localhost/ (or using the host ip, or hostname) I get a 404, which is what I would expect based on the nginx.conf response for /. So that tells me the ingress point is at least up and running. http://localhost/healthz returns a 200, which again I would expect based on nginx.conf
However, doing curl http://localhost/hello always returns a 503.
In the controller logs I just get:
I1011 20:34:13.274379 8 controller.go:168] "Configuration changes detected, backend reload required"
I1011 20:34:13.406122 8 controller.go:185] "Backend successfully reloaded"
I1011 20:34:13.407450 8 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5979dd67b-lrg5r", UID:"49499847-4f0f-4196-ba71-81980445b46c", APIVersion:"v1", ResourceVersion:"444384", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W1011 20:34:21.685551 8 reflector.go:424] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: the server could not find the requested resource
E1011 20:34:21.686258 8 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: the server could not find the requested resource
I1011 20:34:30.136621 8 store.go:430] "Found valid IngressClass" ingress="default/hello-world-ingress" ingressclass="nginx"
W1011 20:34:30.137527 8 endpointslices.go:81] Error obtaining Endpoints for Service "default/hello-world-service": no object matching key "default/hello-world-service" in local store
W1011 20:34:30.137569 8 controller.go:1112] Service "default/hello-world-service" does not have any active Endpoint.
I1011 20:34:30.137664 8 controller.go:168] "Configuration changes detected, backend reload required"
I1011 20:34:30.142794 8 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"hello-world-ingress", UID:"3d297335-e2eb-46cf-a6a5-56bf68725c8f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"447873", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I1011 20:34:30.250894 8 controller.go:185] "Backend successfully reloaded"
I1011 20:34:30.251447 8 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-5979dd67b-lrg5r", UID:"49499847-4f0f-4196-ba71-81980445b46c", APIVersion:"v1", ResourceVersion:"444384", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I1011 20:34:58.695644 8 status.go:299] "updating Ingress status" namespace="default" ingress="hello-world-ingress" currentValue=[] newValue=[{IP:192.168.10.100 Hostname: Ports:[]}]
W1011 20:34:58.702687 8 endpointslices.go:81] Error obtaining Endpoints for Service "default/hello-world-service": no object matching key "default/hello-world-service" in local store
W1011 20:34:58.702735 8 controller.go:1112] Service "default/hello-world-service" does not have any active Endpoint.
I1011 20:34:58.703331 8 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"hello-world-ingress", UID:"3d297335-e2eb-46cf-a6a5-56bf68725c8f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"447926", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W1011 20:34:59.764277 8 reflector.go:424] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: the server could not find the requested resource
E1011 20:34:59.764372 8 reflector.go:140] k8s.io/client-go@v0.25.2/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: the server could not find the requested resource
10.42.0.206 - - [11/Oct/2022:20:35:19 +0000] "GET /hello HTTP/1.1" 503 190 "-" "curl/7.61.1" 78 0.000 [default-hello-world-service-80] [] - - - - 3cc8c307cea5d96642b37e10a01707f1
Doesn't matter if I add a rewrite-target of /, /%2, etc. I've tried defining a type: NodePort for my service, or leaving that off. Doesn't change anything. Seems no matter what the various other problems people had, I can't seem to get this to work.
I've also tried execing into various containers to hit ips of the service, service name, all that, and that seems fine, it's just the path between the ingress point and the service is failing. Turning on debug in my ingress-nginx deployment didn't provide any details that pointed to the issue (at least that I could fathom).
Would greatly appreciate any help that might direct me to a solution, or least help track down where the breakage is occurring. Logs in the actual hello-world pod show successful requests when I run curl from within a pod, but show nothing when I get the 503's, so the request isn't even hitting the service.
I had the same problem with v1.4.0 Rolling back to v1.3.1 fixed the problem for me. Might wanna give it a try.