[SOLVED] Kubernetes load balancer in GKE is not accesible with curl

Kubernetes load balancer in GKE is not accesible with curl

I'm debugging an application running on a Dev environment running in Google Kubernetes Engine and want to send curl based HTTP requests to the LoadBalancer which selects some pods.

$ kubectl get svc -n mynamespace -o wide 
NAME       TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)
mylb   LoadBalancer         1.1.1.1      2.2.2.2   1111:22222/TCP

So as far as I understand CLUSTER-IP is used for access from within the Kubernetes cluster itself while EXTERNAL-IP is used for access from the outside. So I tried the following

$curl -X GET https://2.2.2.2:1111/path
curl: (35) error:02FFF036:system library:func(4095):Connection reset by peer

So this request does not seem to reach the POD behind the LoadBalancer.

Maybe I completely misused the concept of service and LoadBalancer? What is the proper way to send requests to PODs behind a LoadBalancer?

Is there a way to trace requests coming to the LoadBalancer and log which POD it was forwarded (or not forwarded)?

Solution

I would start with looking at your service logs. See if you can find anything there.

If you don't see any traffic there, you could try enabling VPC flow Logs. Allow a little time after enabling, try to curl again and see what shows up in the flow logs.