After upgrading AWS EKS (Kubernetes) to v1.23 from v1.22, Persistent Volumes (PVs) started failing to mount and pods got stuck in pending state

Problem:

I upgraded from AWS EKS cluster to v1.23 from v1.22 and all of a sudden, all the pods that had Persistent Volume Claim (PVC) and Persistent Volume (PV) started failing with the errors like FailedAttachVolume AttachVolume.Attach failed and FailedMount MountVolume.WaitForAttach failed for the AWS EBS volumes.

Pods were giving the following error: Unable to attach or mount volumes timed out waiting for the condition

Solutions Tried:

• I tried adding AWS EBS CSI Driver add-on in the AWS EKS cluster but still it didn't work.
• I tried removing the annotation for migration to this new provisioner on PVCs but that didn't work either.
• I also tried adding storage class for gp3 AWS EBS volume type with AWS EBS CSI Driver as the new provisioner as I was using gp2 but that didn't work either.

Note: AWS EBS volumes were of type gp2.

After adding AmazonEBSCSIDriverPolicy AWS IAM policy to the AWS IAM role that is attached to all the AWS EKS nodes (AWS EC2 instances) and then adding the AWS EBS CSI Driver add-on in the AWS EKS cluster, errors were resolved and PVC got attached successfully. I don't see any issues related to Persistent Volume Claims (PVC), Persistent Volumes (PV), AWS EBS volumes, and pods anymore.

Note: I already had an AWS IAM OpenID Connect (OIDC) provider for my AWS EKS cluster which is a prerequisite for this. In your case, there could be some other issue and the resolution steps might differ so please check out the following reference document: How do I troubleshoot issues with my EBS volume mounts in Amazon EKS?