GitLab Runner AWS S3 Cache is not working - Enable AWS Access

We recently upgraded GitLab and GitLab Runner to version 16.x and tried implementing AWS S3 Cache for caching but that doesn't work. I followed this official documentation here. We are running our GitLab Runners on AWS EKS Kubernetes cluster using Helm.

Following is the error that we see in the jobs:

No URL provided, cache will not be uploaded to shared cache server. Cache will be stored only locally.

This is how our GitLab Runner config looks like:

runners:
  config: |
    [[runners]]
      request_concurrency = 1
      [runners.kubernetes]
        privileged = true
        namespace = "_REDACTED_"
        image = "ubuntu:22.04"
        [runners.kubernetes.node_selector]
        project = "_REDACTED_"
      [runners.cache]
        Type = "s3"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "s3.amazonaws.com"
          BucketName = "_REDACTED_"
          BucketLocation = "us-east-1"
          AuthenticationType = "iam"
          ServerSideEncryption = "S3"
          Insecure = false

Our GitLab Runner node (AWS EC2 instance) already has an AWS IAM role and we also have Service, Role and Role Binding on Kubernetes for our GitLab Runner pod which are created by setting the following in the values file for Helm:

rbac:
  create: true

Our jobs are able to access AWS using the node (AWS EC2 instance) role but for some reason, it is not working with AWS S3. OIDC is also enabled on our AWS EKS Kubernetes cluster.

Note: REDACTED refers to confidential value that has been censored and should be replaced in actual use.

After doing some research, I found the solution. We need to create a separate AWS IAM role for the GitLab Runner pod which will be associated to the Service Account in Kubernetes.

Following is the trust relationship policy for the AWS IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::[AWS_ACCOUNT_ID]:oidc-provider/oidc.eks.[AWS_ACCOUNT_REGION].amazonaws.com/id/[AWS_EKS_OIDC_ID]"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.[AWS_ACCOUNT_REGION].amazonaws.com/id/[AWS_EKS_OIDC_ID]:sub": "system:serviceaccount:[AWS_EKS_KUBERNETES_NAMESPACE_NAME]:[AWS_EKS_KUBERNETES_SERVICE_ACCOUNT_NAME]",
                    "oidc.eks.[AWS_ACCOUNT_REGION].amazonaws.com/id/[AWS_EKS_OIDC_ID]:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Following is the permissions policy for the AWS IAM role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::[AWS_S3_BUCKET_NAME]",
                "arn:aws:s3:::[AWS_S3_BUCKET_NAME]/*",
            ]
        }
    ]
}

Note: All the values between [...] are supposed to be replaced by correct actual values.

Following are the changes made in the values file for Helm:

rbac:
  create: true
  serviceAccountAnnotations:
    eks.amazonaws.com/role-arn: "_REDACTED_"

References:

• GitLab - Issues - AWS IAM Authentication Method not working since Gitlab runner v16
• GitLab - Issues - ERROR: error while generating S3 pre-signed URL error=Not authorized to perform sts:AssumeRoleWithWebIdentity after upgrading Helm chart
• AWS Documentation - IAM roles for service accounts
• AWS Documentation - Configuring a Kubernetes service account to assume an IAM role
• GitLab Documentation - GitLab Runner - Advanced Configuration - The [runners.cache.s3] section

Note: REDACTED refers to confidential value that has been censored and should be replaced in actual use.