kubernetespersistent-volumesrook-storagekubernetes-rook

What node field means in k8s volume.attachments?


I have the following Volume:

Name:         pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
Namespace:    rook-ceph-system
Labels:       <none>
Annotations:  <none>
API Version:  rook.io/v1alpha2
Attachments:
  Cluster Name:   rook-ceph
  Mount Dir:      /var/lib/kubelet/pods/72fd4f89-5110-49b7-8d88-87488b58695c/volumes/ceph.rook.io~rook-ceph-system/pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
  Node:           node-6.xyz.com
  Pod Name:       dev-cockroachdb-0
  Pod Namespace:  x-namespace
  Read Only:      false
Kind:             Volume
Metadata:
  Creation Timestamp:  2020-08-12T17:13:51Z
  Generation:          6
  Resource Version:    638003207
  Self Link:           /apis/rook.io/v1alpha2/namespaces/rook-ceph-system/volumes/pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
  UID:                 db0a9491-95fe-49cd-8160-89031847d636
Events:                <none>

For the pod dev-cockroachdb-0 I'm getting the following error:

MountVolume.SetUp failed for volume "pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738 for pod x-namespace/dev-cockroachdb-0. Volume is already attached by pod x-namespace/dev-cockroachdb-0. Status Pending

And the pod x-namespace/dev-cockroachdb-0 is currently scheduled to node-5.xyz.com.

So, as you can see the pod itself is in a different node than the VolumeAttachment. node-6.xyz.com vs node-5.xyz.com.

Questions:


Solution

  • You didn't mention it but it looks like you have a Rook volume. Perhaps a pvc-clone, something like this:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: rbd-pvc-clone
    spec:
      storageClassName: rook-ceph-block
      dataSource:
        name: rbd-pvc
        kind: PersistentVolumeClaim
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
    

    Does Node in Volume.Attachments point to the node on which the pod (to which the volume is attached) is located? (So if the volume is attached to a pod on node NodeA, then the value of node field for the volume attachment will be NodeA)

    Not really, your Pod may be trying to start on node-5.xyz.com, but the volume could be attached to another node in your case it's node-6.xyz.com.

    May this error happen because of the failure to correctly detach the volume on some node?

    Yes, this error may happen if maybe you had another pod running in node-6.xyz.com that terminated and failed to detach.

    Keep in mind that all of the above is considering that your volume has accessModes: ReadWriteOnce. It looks like you are using Ceph with Rook and in that case, you can also use accessModes: ReadWriteMany which would essentially allow you to attach the volume to Kubernetes nodes.

    ✌️