GKE k8 gateway api route not working saying "no healthy upstream" even when the gateway and routes are created healthy .Below are the configs
apiVersion: networking.gke.io/v1
kind: GCPGatewayPolicy
metadata:
name: monitoring-gateway-policy
spec:
default:
allowGlobalAccess: true
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: monitoring-gateway
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: monitoring-gateway
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/tls-acme: "true"
spec:
gatewayClassName: gke-l7-rilb
listeners:
- name: https
protocol: HTTPS
hostname: abc.com
port: 443
tls:
mode: Terminate
certificateRefs:
- kind: Secret
group: ""
name: monitoring-ingress-tls
allowedRoutes:
namespaces:
from: All
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: monitoring-routes
spec:
parentRefs:
- kind: Gateway
name: monitoring-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /grafana
backendRefs:
- name: internal-grafana
namespace: monitoring
port: 3000
- matches:
- path:
type: PathPrefix
value: /chronograf
backendRefs:
- name: chronograf
namespace: monitoring
port: 80
- backendRefs:
- name: jaeger-query
port: 80
matches:
- path:
type: PathPrefix
value: /jaeger
even the gateway and routes are healthy and pointing to the service , which has endpoints to the pods . abc.com/grafana , abc.com/chronograf , abc.com/jaeger does not work and says "no healthy upstream"
The "no healthy upstream" error can be caused by several factors, most commonly related to failing health checks, missing or unhealthy endpoints, port mismatches, or general misconfigurations that prevent back-ends services from being reachable.
Start by verifying the health check endpoints for chronograf, internal-grafana and jaeger-query in the Google Cloud Console and ensure they return HTTP 200.
Service port mismatch, check the HTTPRoute specified ports for the backend services 3000 for internal-grafana, 80 for chronograf and 80 for jaeger-query. If these ports don’t match the actual ports exposed by the Service or the pod’s container, the gateway cannot route traffic correctly.
Run the following commands to validate:
kubectl get pods -n monitoring
kubectl describe svc <service-name> -n monitoring
kubectl describe endpoints <service-name> -n monitoring
Confirm that the Endpoints object for each service lists valid pod IPs. Ensure the backend pods are running and reachable.
Use the command below to confirm that each pod is listening on the correct port:
kubectl describe pod <grafana-pod-name> -n monitoring
Ensure the pod’s container is listening on the specified port example Grafana listens on 3000, Chronograf on 80, etc.
I also highly recommend to include the output of these kubectl commands in your post or comments to assist with further debugging.
For further reference you can refer to Understanding No Healthy Upstream Error.