kubernetesgoogle-cloud-platformgoogle-kubernetes-enginekubectlgke-networking

GCP kubernetes trying to access service through apiserver proxy URLs


I'm trying to access a service in my cluster from my laptop. I have tried to use this solution https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster-services/.

After running kubectl proxy --port=8080 I'm trying to access my service through this endpoint: http://127.0.0.1:8080/api/v1/namespaces/myns/services/mysvc/proxy/health-check But I'm receiving always :

{
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "message": "error trying to reach service: dial tcp ip3:8080: i/o timeout",
    "reason": "ServiceUnavailable",
    "code": 503
}

There are endpoints that it's working like this one: http://127.0.0.1:8080/api/v1/namespaces/myns/services/mysvc

But when I'm trying to use the /proxy it always returns 503

My service:
Name:                     mysvc
Namespace:                myns
Labels:                   app.kubernetes.io/instance=myns-mysvc-dev
Annotations:              cloud.google.com/load-balancer-type: Internal
Selector:                 app=mysvc
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       ip1
IPs:                      ip1
LoadBalancer Ingress:     ip2
Port:                     unset  443/TCP
TargetPort:               8080/TCP
NodePort:                 unset  30840/TCP
Endpoints:                ip3:8080
Session Affinity:         None
External Traffic Policy:  Cluster

Solution

  • As per this official doc,

    The allowed ports (443 and 10250) refer to the ports exposed by your nodes and Pods, not the ports exposed by any Kubernetes services. For example, if the cluster control plane attempts to access a service on port 443, but the service is implemented by a pod using port 9443, this will be blocked by the firewall unless you add a firewall rule to explicitly allow ingress to port 9443.

    So, this is an intended behavior that only allows ports 443 (https) and 10250 (kubelet) by default for master-to-node traffic. In your case, the service is not reaching the 8080 port. So you need to add this firewall rule that should accept the traffic from the 8080 port. You can add additional firewall rules if needed by using this Adding firewall rules for specific use cases .

    You can also check your security group configuration. Ensure the control plane can access worker nodes on required ports. So that this proxy might work on the 8080 port also. Refer to this issue for more information