I have a kubernetes cluster spread across two zones, A and B. I am using nfs volumes for persistent storage. I have nfs volumes in both the zones. I am creating a stateful set of 2 replicas which will be spread across these zones (I used pod anti-affinity to achieve this). Now I want the pods in zone A to use the volumes in zone A and ones in zone B to use the volumes in zone B.
I can add labels to the persistent volumes and match the persistent volume claims with these labels. But how do I make sure that the pvc for a pod does not get bound to a pv in another zone?
You can try to bind persistent volume claims (PVCs) to persistent volumes (PVs) and split Kubernetes pods across your cluster between two zones using the special built-in label failure-domain.beta.kubernetes.io/zone. If you create volumes manually, it is possible to label them with failure-domain.beta.kubernetes.io/zone:zoneA value, ensuring that pod is only scheduled to nodes in the same zone as the zone of the persistent volume.
For example, to set label for a Node and PV:
kubectl label node <node-name> failure-domain.beta.kubernetes.io/zone=zoneA
kubectl label pv <pv-name> failure-domain.beta.kubernetes.io/zone=zoneA
Find some useful information from official Kubernetes documentation.