kubernetesredisheadlessheadless-service

Kubernetes headless service endpoint problem

We have a stateful redis deployment on multi node Kubernetes cluster. (v1.27.15)

There are two services named "redis" and "redis-headless"

There are 3 nodes in the cluster. When we shutdown one of the nodes, redis on that node becomes terminating:

kubectl get pods -A -o wide | grep redis
mynamespace redis-node-0  3/3     Running     0   8m  10.244.248.4    ha3-node2
mynamespace redis-node-1  3/3     Terminating 0  68m  10.244.230.119  ha3-node1
mynamespace redis-node-2  3/3     Running     0  67m  10.244.192.208  ha3-node3

But for redis-headless service 10.244.230.119 is still in endpoints

kubectl describe endpoints  -n mynamespace redis-headless
Name:         redis-headless
Namespace:    mynamespace
Subsets:
  Addresses:          10.244.192.208,10.244.230.119,10.244.248.4

For redis service (clusterIP) endpoints are OK. (10.244.230.119 is deleted from endpoints)

Is this behaviour normal for headless service, if not what is the solution?

Regards,

Yavuz

Solution

  • The Headless service likely has publishNotReadyAddresses set to true in the service manifest. If this is true, then the IP of that terminating pod can still be shown in the endpoint resource until it is fully terminated. Once the controller recreates the pod, the new IP will show.

    After all, the Headless service does not have the kube-proxy handle it. The client can directly connect to the Pods via Cluster DNS - that is the whole point.

    So it is normal in my opinion based on the above.