amazon-web-serviceskubernetes

AWS EKS secrets store provider - Failed to fetch secret from all regions: name


I am using the AWS secrets store CSI provider to sync secrets from the AWS Secret Manager into Kubernetes/EKS.

The SecretProviderClass is:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: test-provider
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: mysecret
        objectType: secretsmanager
        jmesPath:
          - path: APP_ENV
            objectAlias: APP_ENV
          - path: APP_DEBUG
            objectAlias: APP_DEBUG

And the Pod mounting these secrets is:

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
spec:
  restartPolicy: Never
  serviceAccountName: my-account
  terminationGracePeriodSeconds: 2
  containers:
    - name: dotfile-test-container
      image: registry.k8s.io/busybox
      volumeMounts:
        - name: secret-volume
          readOnly: true
          mountPath: "/mnt/secret-volume"
  volumes:
    - name: secret-volume
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: test-provider

The secret exists in the Secret Provider:

{
  "APP_ENV": "staging",
  "APP_DEBUG": false
}

(this is an example, I am aware I do not need to store these particular variables as secrets)

But when I create the resources, the Pod fails to run with

Warning  
FailedMount  
96s (x10 over 5m47s)  
kubelet            

MountVolume.SetUp failed for volume "secret-volume" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod pace/secret-dotfiles-pod, 
err: rpc error: code = Unknown desc = Failed to fetch secret from all regions: mysecret

Solution

  • Turns out the error message is very misleading. The problem in my case was due to the type of the APP_DEBUG value. Changing it from a boolean to string fixed the problem and now the pod starts correctly.

    {
      "APP_ENV": "staging",
      "APP_DEBUG": "false"
    }
    

    Seems like a bug in the provider to me.