I have the following Volume
:
Name: pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
Namespace: rook-ceph-system
Labels: <none>
Annotations: <none>
API Version: rook.io/v1alpha2
Attachments:
Cluster Name: rook-ceph
Mount Dir: /var/lib/kubelet/pods/72fd4f89-5110-49b7-8d88-87488b58695c/volumes/ceph.rook.io~rook-ceph-system/pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
Node: node-6.xyz.com
Pod Name: dev-cockroachdb-0
Pod Namespace: x-namespace
Read Only: false
Kind: Volume
Metadata:
Creation Timestamp: 2020-08-12T17:13:51Z
Generation: 6
Resource Version: 638003207
Self Link: /apis/rook.io/v1alpha2/namespaces/rook-ceph-system/volumes/pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738
UID: db0a9491-95fe-49cd-8160-89031847d636
Events: <none>
For the pod dev-cockroachdb-0
I'm getting the following error:
MountVolume.SetUp failed for volume "pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738" : mount command failed, status: Failure, reason: Rook: Mount volume failed: failed to attach volume pvc-c8a0c1ee-b9e6-11e9-9ffa-0cc47ab04738 for pod x-namespace/dev-cockroachdb-0. Volume is already attached by pod x-namespace/dev-cockroachdb-0. Status Pending
And the pod x-namespace/dev-cockroachdb-0
is currently scheduled to node-5.xyz.com
.
So, as you can see the pod itself is in a different node than the VolumeAttachment
. node-6.xyz.com
vs node-5.xyz.com
.
Questions:
Node
in Volume.Attachments
point to the node on which the pod (to which the volume is attached) is located? (So if the volume is attached to a pod on node NodeA
, then the value of node field for the volume attachment will be NodeA
)You didn't mention it but it looks like you have a Rook volume. Perhaps a pvc-clone, something like this:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-clone
spec:
storageClassName: rook-ceph-block
dataSource:
name: rbd-pvc
kind: PersistentVolumeClaim
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Does Node in Volume.Attachments point to the node on which the pod (to which the volume is attached) is located? (So if the volume is attached to a pod on node NodeA, then the value of node field for the volume attachment will be NodeA)
Not really, your Pod may be trying to start on node-5.xyz.com
, but the volume could be attached to another node in your case it's node-6.xyz.com
.
May this error happen because of the failure to correctly detach the volume on some node?
Yes, this error may happen if maybe you had another pod running in node-6.xyz.com
that terminated and failed to detach.
Keep in mind that all of the above is considering that your volume has accessModes: ReadWriteOnce
. It looks like you are using Ceph with Rook and in that case, you can also use accessModes: ReadWriteMany
which would essentially allow you to attach the volume to Kubernetes nodes.
✌️