azurekubernetesazure-aksazure-keyvaultaad-pod-identity

Azure Key Vault integration with AKS works for nginx tutorial Pod, but not actual project deployment


Per the title, I have the integration working following the documentation.

I can deploy the nginx.yaml and after about 70 seconds I can print out secrets with:

kubectl exec -it nginx -- cat /mnt/secrets-store/secret1

Now I'm trying to apply it to a PostgreSQL deployment for testing and I get the following from the Pod description:

  Warning  FailedMount             3s    kubelet                  MountVolume.SetUp failed for volume "secrets-store01-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod staging/postgres-deployment-staging-69965ff767-8hmww, err: rpc error: code = Unknown desc = failed to mount objects, error: failed to get keyvault client: failed to get key vault token: nmi response failed with status code: 404, err: <nil>

And from the nmi logs:

E0221 22:54:32.037357       1 server.go:234] failed to get identities, error: getting assigned identities for pod staging/postgres-deployment-staging-69965ff767-8hmww in CREATED state failed after 16 attempts, retry duration [5]s, error: <nil>. Check MIC pod logs for identity assignment errors
I0221 22:54:32.037409       1 server.go:192] status (404) took 80003389208 ns for req.method=GET reg.path=/host/token/ req.remote=127.0.0.1

Not sure why since I basically copied the settings from the nignx.yaml into the postgres.yaml. Here they are:

# nginx.yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx
  namespace: staging
  labels:
    aadpodidbinding: aks-akv-identity-binding-selector
spec:
  containers:
    - name: nginx
      image: nginx
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: /mnt/secrets-store
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: aks-akv-secret-provider
# postgres.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-deployment-staging
  namespace: staging
  labels:
    aadpodidbinding: aks-akv-identity-binding-selector
spec:
  replicas: 1
  selector:
    matchLabels:
      component: postgres
  template:
    metadata:
      labels:
        component: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:13-alpine
          ports:
            - containerPort: 5432
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: /mnt/secrets-store
            readOnly: true
          - name: postgres-storage-staging
            mountPath: /var/postgresql
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: aks-akv-secret-provider
        - name: postgres-storage-staging
          persistentVolumeClaim:
            claimName: postgres-storage-staging

---
apiVersion: v1
kind: Service
metadata:
  name: postgres-cluster-ip-service-staging
  namespace: staging
spec:
  type: ClusterIP
  selector:
    component: postgres
  ports:
    - port: 5432
      targetPort: 5432

Suggestions for what the issue is here?


Solution

  • Oversight on my part... the aadpodidbinding should be in the template: per:

    https://azure.github.io/aad-pod-identity/docs/best-practices/#deploymenthttpskubernetesiodocsconceptsworkloadscontrollersdeployment

    The resulting YAML should be:

    # postgres.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: postgres-deployment-production
      namespace: production
    spec:
      replicas: 1
      selector:
        matchLabels:
          component: postgres
      template:
        metadata:
          labels:
            component: postgres
            aadpodidbinding: aks-akv-identity-binding-selector
        spec:
          containers:
            - name: postgres
              image: postgres:13-alpine
              ports:
                - containerPort: 5432
              env: 
                - name: POSTGRES_DB_FILE
                  value: /mnt/secrets-store/DEV-PGDATABASE
                - name: POSTGRES_USER_FILE
                  value: /mnt/secrets-store/DEV-PGUSER
                - name: POSTGRES_PASSWORD_FILE
                  value: /mnt/secrets-store/DEV-PGPASSWORD
                - name: POSTGRES_INITDB_ARGS
                  value: "-A md5"
                - name: PGDATA
                  value: /var/postgresql/data
              volumeMounts:
              - name: secrets-store01-inline
                mountPath: /mnt/secrets-store
                readOnly: true
              - name: postgres-storage-production
                mountPath: /var/postgresql
          volumes:
            - name: secrets-store01-inline
              csi:
                driver: secrets-store.csi.k8s.io
                readOnly: true
                volumeAttributes:
                  secretProviderClass: aks-akv-secret-provider
            - name: postgres-storage-production
              persistentVolumeClaim:
                claimName: postgres-storage-production
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: postgres-cluster-ip-service-production
      namespace: production
    spec:
      type: ClusterIP
      selector:
        component: postgres
      ports:
        - port: 5432
          targetPort: 5432