Gitlab runner not able to execute helm list command

I'm trying to setup a gitlab kubernetes agent and runner for my in-cluster CICD pipeline. My gitlab.ci is something on the line of:

stages:
  - deploy

deploy-new-images:
  stage: deploy
  image:
    name: alpine/helm:3.7.1
    entrypoint: [""]
  script:
    - helm list --all-namespaces
  tags:
    - staging
    - test

Gitlab is able to start the container for this particular job but fails with the following error:

Error: list: failed to list: secrets is forbidden: User "system:serviceaccount:gitlab:default" cannot list resource "secrets" in API group "" at the cluster scope

My agent deployment has a service account named gitlab. This service account is associated to the default admin ClusterRole via a ClusterRoleBinding; it is also associated with a Role via a RoleBinding resource. The role has the following definition :

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: "2022-04-14T15:27:27Z"
  labels:
    app: gitlab-agent
  name: gitlab-agent
  namespace: gitlab
  resourceVersion: "44732"
  uid: 0b09cabe-826f-4c7c-a864-3192e0c9ea42
rules:
- apiGroups:
  - ""
  resources:
  - '*'
  verbs:
  - '*'

The gitlab runner on the other hand was deployed via Helm using the following values:

  image: "gitlab/gitlab-runner:alpine-v14.9.0"
  imagePullPolicy: "Always"
  replicas: 5
  gitlabUrl: https://gitlab.com
  runnerRegistrationToken: "{{ vault_gitlab_runner_registration_token }}"
  unregisterRunners: true
  logLevel: "debug"
  rbac:
    create: true
    rules:
    - resources: ["*"]
      verbs: ["*"]
    - apiGroups: [""]
      resources: ["*"]
      verbs: ["*"]
  clusterWideAccess: true
  metrics:
    enabled: false
  runners:
    executor: kubernetes
    locked: false
    tags: "staging,test"
    name: "test-staging-gitlab-runner"
    runUntagged: false
    config: |
      [[runners]]
        [runners.kubernetes]
          namespace = "gitlab"
          image = "ubuntu:20.04"

It seems to me that both the agent and the runner have sufficient permissions and yet the helm list command is failing. Any idea?

In short: In your gitlab-runner values file, add the setting service_account = "gitlab" under [runners.kubernetes].

Explanation: The actual pod that executes the job is not the gitlab-runner pod, nor the gitlab agent pod. Gitlab runner kubernetes executor spawns a new pod for each job. The place to set the service account name for these pods is in the above setting.

The error message above shows that the pod running the job was using the default service account for the gitlab namespace. this service account doesn't (and shouldn't) have the required permissions.

The section runners.config in the values file, ends up in the config.toml file that configures the runner. Here is the documentation about config.toml for kubernetes executor: https://docs.gitlab.com/runner/executors/kubernetes.html#other-configtoml-settings

And here is the Kubernetes executor interaction diagram