PersistentVolumeClaim is stuck 'waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com"' on new AWS EKS cluster

We have a EKS setup provisioned where we use a Cloud Native Buildpacks based Tekton Pipeline our PipelineRuns are stuck and pending forever without getting a Pod resource. We created a PersistentVolumeClaim like this:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: buildpacks-source-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi

Having a look into the events of this PVC after creation there are the following events indicating something is wrong with our EKS setup:

The command kubectl describe pvc buildpacks-source-pvc gives the following event messages:

Name:          buildpacks-source-pvc
Namespace:     default
StorageClass:  gp2
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
               volume.kubernetes.io/selected-node: ip-999-99-99-99.eu-central-1.compute.internal
               volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       affinity-assistant-0b3d266b91-0
               affinity-assistant-53a7c08baf-0
               affinity-assistant-a375f28de3-0
               affinity-assistant-e8cb1a6e15-0
               buildpacks-test-pipeline-run-9rz4l-fetch-repository-pod
Events:
  Type    Reason                Age                     From                         Message
  ----    ------                ----                    ----                         -------
  Normal  ExternalProvisioning  3m43s (x561 over 143m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

What is this EBS CSI thing and how do we get our cluster working as before?

From EKS 1.23 on a Container Storage Interface (CSI) driver is needed in order to get your PersisentVolumeClaims served by a PersistentVolume as you are used to from earlier EKS versions.

The docs tell us, what needs to be configured:

Solution: Configure Amazon EBS CSI driver for working PersistentVolumes in EKS

In essence we need to enable the AWS EBS CSI driver as an EKS addon. But beforehand we need to enable the IAM OIDC provider and create the IAM role for the EBS CSI driver. The easiest way to do both is to use eksctl (other ways like using plain aws cli or the AWS GUI are described in the docs).

1.) Install eksctl

We assume here that the aws cli is installed and configured - and you have access to your EKS cluster. To use eksctl we need to install it first. On a Mac use brew like:

brew tap weaveworks/tap
brew install weaveworks/tap/eksctl

or on Linux use:

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin

2.) Enable IAM OIDC provider

A prerequisite for the EBS CSI driver to work is to have an existing AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for your cluster. This IAM OIDC provider can be enabled with the following command:

eksctl utils associate-iam-oidc-provider --region=eu-central-1 --cluster=YourClusterNameHere --approve

3.) Create Amazon EBS CSI driver IAM role

Now having eksctl in place, create the IAM role:

eksctl create iamserviceaccount \
  --region eu-central-1 \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster YourClusterNameHere \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --approve \
  --role-only \
  --role-name AmazonEKS_EBS_CSI_DriverRole

As you can see AWS maintains a managed policy for us we can simply use (AWS maintains a managed policy, available at ARN arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy). Only if you use encrypted EBS drives you need to additionally add configuration to the policy.

The command...

...deploys an AWS CloudFormation stack that creates an IAM role, attaches the IAM policy to it, and annotates the existing ebs-csi-controller-sa service account with the Amazon Resource Name (ARN) of the IAM role.

4.) Add the Amazon EBS CSI add-on

Now we can finally add the EBS CSI add-on. Therefor we also need the AWS Account id which we can obtain by running aws sts get-caller-identity --query Account --output text (see Quick way to get AWS Account number from the AWS CLI tools?). Now the eksctl create addon command looks like this:

eksctl create addon --name aws-ebs-csi-driver --cluster YourClusterNameHere --service-account-role-arn arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):role/AmazonEKS_EBS_CSI_DriverRole --force

Now your PersistentVolumeClaim should get the status Bound while a EBS volume got created for you - and the Tekton Pipeline should run again.