Since I am a newbie in Kubernetes and trying to work with it I am getting an error of No agent available while executing
kubectl exec -it <POD NAME> -c <CONTAINER NAME> -- /bin/bash
# Or
kubectl logs po/<POD NAME>
Full error
error: Internal error occurred: error sending request: Post "https://172.24.2.61:10250/exec/default/backend-595fbf9667-h2bzx/backend-sha256-1?command=%!F(MISSING)bin%!F(MISSING)bash&input=1&output=1&tty=1": No agent available
Or
Error from server: Get "https://172.24.2.61:10250/containerLogs/utshab/frontend-5969767748-prlkn/frontend": No agent available
Cluster information:
Kubernetes version:
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.30.5-gke.1699000
Cloud being used: GCP
Installation method: Managed cluster
Host OS: Container-Optimized OS from Google
CNI and version: Calico v3.26.3-gke.17
CRI and version: containerd 1.7.23
Any one if can guide me why am I getting this error and what could be the solution to it, would be much appreciated.
Few other info for reference
This is happening for all pods
Getting the same error for all other bash commands such as (date, pwd)
Some more information
Upon executing with verbose I see an HTTP 500 error.
kubectl logs po/counter-err -v=9
I0223 10:43:48.164582 272521 loader.go:395] Config loaded from file: /home/utshab_saha_accenture_com/.kube/config
I0223 10:43:48.171614 272521 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.31.0 (linux/amd64) kubernetes/9edcffc" 'https://172.24.5.2/api/v1/namespaces/default/pods/counter-err'
I0223 10:43:48.277606 272521 round_trippers.go:510] HTTP Trace: Dial to tcp:172.24.5.2:443 succeed
I0223 10:43:48.350996 272521 round_trippers.go:553] GET https://172.24.5.2/api/v1/namespaces/default/pods/counter-err 200 OK in 179 milliseconds
I0223 10:43:48.351080 272521 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 99 ms TLSHandshake 3 ms ServerProcessing 69 ms Duration 179 ms
I0223 10:43:48.351116 272521 round_trippers.go:577] Response Headers:
I0223 10:43:48.351140 272521 round_trippers.go:580] Content-Type: application/json
I0223 10:43:48.351159 272521 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: a6feede1-df10-495d-8799-8d405b6b784e
I0223 10:43:48.351176 272521 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: c625a2a4-2f7f-4ce6-92b9-6a012e472d90
I0223 10:43:48.351192 272521 round_trippers.go:580] Date: Sun, 23 Feb 2025 10:43:48 GMT
I0223 10:43:48.351207 272521 round_trippers.go:580] Audit-Id: d6ff0ab0-1fe3-4029-bca2-9b767452c752
I0223 10:43:48.351222 272521 round_trippers.go:580] Cache-Control: no-cache, private
I0223 10:43:48.351420 272521 request.go:1351] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"counter-err","namespace":"default","uid":"06349c11-f889-4178-83e9-c2ced99e29db","resourceVersion":"339340855","creationTimestamp":"2025-02-23T10:34:53Z","annotations":{"cni.projectcalico.org/containerID":"9ad31efb9e03f3ef53071a17f624f55e1acfb59c33dd8a424a1063448096aba0","cni.projectcalico.org/podIP":"172.24.8.40/32","cni.projectcalico.org/podIPs":"172.24.8.40/32","kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Pod\",\"metadata\":{\"annotations\":{},\"name\":\"counter-err\",\"namespace\":\"default\"},\"spec\":{\"containers\":[{\"args\":[\"/bin/sh\",\"-c\",\"i=0; while true; do echo \\\"$i: $(date)\\\"; echo \\\"$i: err\\\" \\u003e\\u00262 ; i=$((i+1)); sleep 1; done\"],\"image\":\"busybox:1.28\",\"name\":\"count\"}]}}\n"},"managedFields":[{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2025-02-23T10:34:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"count\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2025-02-23T10:34:56Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/containerID":{},"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}},"subresource":"status"},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2025-02-23T10:34:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodReadyToStartContainers\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:hostIPs":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"172.24.8.40\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]},"spec":{"volumes":[{"name":"kube-api-access-fxqxc","projected":{"sources":[{"serviceAccountToken":{"expirationSeconds":3607,"path":"token"}},{"configMap":{"name":"kube-root-ca.crt","items":[{"key":"ca.crt","path":"ca.crt"}]}},{"downwardAPI":{"items":[{"path":"namespace","fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}]}}],"defaultMode":420}}],"containers":[{"name":"count","image":"busybox:1.28","args":["/bin/sh","-c","i=0; while true; do echo \"$i: $(date)\"; echo \"$i: err\" \u003e\u00262 ; i=$((i+1)); sleep 1; done"],"resources":{},"volumeMounts":[{"name":"kube-api-access-fxqxc","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"gke-genai-private-gke-private-np-1-46f80ca2-uzxg","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-02-23T10:34:58Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-02-23T10:34:55Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-02-23T10:34:58Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-02-23T10:34:58Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-02-23T10:34:55Z"}],"hostIP":"172.24.2.61","hostIPs":[{"ip":"172.24.2.61"}],"podIP":"172.24.8.40","podIPs":[{"ip":"172.24.8.40"}],"startTime":"2025-02-23T10:34:55Z","containerStatuses":[{"name":"count","state":{"running":{"startedAt":"2025-02-23T10:34:57Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/library/busybox:1.28","imageID":"docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47","containerID":"containerd://ef886d0c80610fa4bbf71bdf524986bc691d8cb6653163d2f5a9e527c035f2d3","started":true}],"qosClass":"BestEffort"}}
I0223 10:43:48.354893 272521 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.31.0 (linux/amd64) kubernetes/9edcffc" 'https://172.24.5.2/api/v1/namespaces/default/pods/counter-err/log?container=count'
I0223 10:43:48.394695 272521 round_trippers.go:553] GET https://172.24.5.2/api/v1/namespaces/default/pods/counter-err/log?container=count 500 Internal Server Error in 39 milliseconds
I0223 10:43:48.394730 272521 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 39 ms Duration 39 ms
I0223 10:43:48.394739 272521 round_trippers.go:577] Response Headers:
I0223 10:43:48.394759 272521 round_trippers.go:580] Cache-Control: no-cache, private
I0223 10:43:48.394777 272521 round_trippers.go:580] Content-Type: application/json
I0223 10:43:48.394813 272521 round_trippers.go:580] Content-Length: 186
I0223 10:43:48.394826 272521 round_trippers.go:580] Date: Sun, 23 Feb 2025 10:43:48 GMT
I0223 10:43:48.394837 272521 round_trippers.go:580] Audit-Id: d5dda5ea-af87-487c-98b6-376feb2f7b96
I0223 10:43:48.394878 272521 request.go:1351] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \"https://172.24.2.61:10250/containerLogs/default/counter-err/count\": No agent available","code":500}
I0223 10:43:48.395293 272521 helpers.go:246] server response object: [{
"metadata": {},
"status": "Failure",
"message": "Get \"https://172.24.2.61:10250/containerLogs/default/counter-err/count\": No agent available",
"code": 500
}]
Error from server: Get "https://172.24.2.61:10250/containerLogs/default/counter-err/count": No agent available
Finally I managed to solve the issue. let me break the ice, the culprit was the Network Firewall.
Now let me explain what happened. The issue relied in the communication between Kube API Server and worker nodes. It was only kubectl exec, logs, port-forward these commands which did not work earlier, all other kubectl worked pretty well. The solution was hidden in the fact how these commands are actually executed.
In contrast to other kubectl commands exec, logs, top, port-forward these works slightly different way. These commands needs direct communication between kubectl client and worker nodes, hence it requires TCP tunnel to be established. And that tunnel is established via Konnectivity agents which are deployed on all worker nodes. This agent establish a connection with kube API Server via a TCP port 8132. Hence this 8132 must be allowed in the egress firewall rule.
So in my case this port was missing in the rules hence all the Konnectivity agent pods were down, meaning no tunnel was established, which signifies the error message No agent available.
Reference - https://cloud.google.com/kubernetes-engine/docs/troubleshooting/kubectl#konnectivity_proxy