I am trying to use EFS as the persistent storage for some EKS pods. I have installed the AWS EFS CSI Driver, created a File System in EFS and a storage class in EKS. When I try to mount a volume to a pod, the PVC fails with the error -
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 19s efs.csi.aws.com_ip-xxxx.xxx.com_7598289c-9f51-4b83-9a4e-02ff9942af9a failed to provision volume with StorageClass "efs-sc": rpc error: code = Internal desc = Failed to fetch File System info: Describe File System failed: WebIdentityErr: failed to retrieve credentials
caused by: ValidationError: Request ARN is invalid
status code: 400, request id: 2af47ad1-c5ce-4389-a716-8f0e241145a7
I am not sure where I am going wrong. I have set up the CSI driver by following the documentation here and have the proper IRSA set up as per the document. I expected that the PVC would be bound and the pod would go in running state but the PVC stays unbound and the pod goes into pending state. This is my first time using EFS and I am kind of clueless right now. Any guidance is really appreciated. Is there any way to dig deeper and understand why it throws the error it does as I have ensured that the fileSystemId in the storage class is the same as the file system ID shown in the EFS console? What might I be missing?
The AWS EFS CSI driver controller logs show the following fileSystemId which is the same as the one in AWS console.
I0729 06:59:09.646688 1 controller.go:61] CreateVolume: called with args {Name:pvc-649527dd-78bf-4aaa-9688-2496bb181d6c CapacityRange:required_bytes:68719476736 VolumeCapabilities:[mount:<mount_flags:"tls" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Parameters:map[basePath:/dynamic_provisioning csi.storage.k8s.io/pv/name:pvc-649527dd-78bf-4aaa-9688-2496bb181d6c csi.storage.k8s.io/pvc/name:prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-2 csi.storage.k8s.io/pvc/namespace:monitoring directoryPerms:755 fileSystemId:fs-08121f8be9526a369 gidRangeEnd:70000 gidRangeStart:1000 provisioningMode:efs-ap] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0729 06:59:09.646782 1 cloud.go:238] Calling DescribeFileSystems with input: {
FileSystemId: "fs-08121f8be9526a369"
}
[EFS console screenshot][1]
[1]: https://i.sstatic.net/S6HNd.png
usually you get ARN invlaid error is when you give incorrect arn names, In the referred document ARN is mainly used in 3 places - One in the seciton - "Principal": "Federated" in the policy and in service acccount metadata annotations, make sure you add the correct arn from your account. In policy search for efs-eks-AmazonEFSCSIDriverPolicyRole and make sure you add the correct arn and replace the accountId
something like
annotations: eks.amazonaws.com/role-arn: arn:aws:iam::YOUR_ACCOUNT_ID:role/COMP/apps/services/entarch/efs-eks-AmazonEFSCSIDriverPolicyRole-REMAINING_VALUES_REPLACE