Error: couldn't find key DATABASE_USER in Secret airbyte-airbyte-secrets

I am trying to deploy Airbyte to Amazon EKS by its helm chart. I am using Airbyte v0.63.1 and Helm chart v0.199.0.

Based on this tutorial, I have created Kubernetes secret production-hm-airbyte-secret in advance with values

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• LOG_S3_BUCKET_NAME
• LOG_S3_BUCKET_REGION
• POSTGRES_USER_NAME
• POSTGRES_PASSWORD

Based on this original values.yaml, here is my-values.yaml:

global:
  state:
    storage:
      type: S3
  database:
    type: external
    host: production-hm-airbyte-postgres.xxx.us-west-2.rds.amazonaws.com
    port: 5432
    database: airbyte_db
    user: airbyte_user
    secretName: production-hm-airbyte-secret
    passwordSecretKey: POSTGRES_PASSWORD
  logs:
    accessKey:
      existingSecret: production-hm-airbyte-secret
      existingSecretKey: AWS_ACCESS_KEY_ID
    secretKey:
      existingSecret: production-hm-airbyte-secret
      existingSecretKey: AWS_SECRET_ACCESS_KEY
  storage:
    type: S3
  minio:
    enabled: false
server:
  extraEnv:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_BUCKET_NAME
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_NAME
    - name: STATE_STORAGE_S3_REGION
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_REGION
worker:
  extraEnv:
    - name: AWS_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: AWS_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_ACCESS_KEY_ID
    - name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: AWS_SECRET_ACCESS_KEY
    - name: STATE_STORAGE_S3_BUCKET_NAME
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_NAME
    - name: STATE_STORAGE_S3_REGION
      valueFrom:
        secretKeyRef:
          name: production-hm-airbyte-secret
          key: LOG_S3_BUCKET_REGION
postgresql:
  enabled: false
externalDatabase:
  host: production-hm-airbyte-postgres.xxx.us-west-2.rds.amazonaws.com
  port: 5432
  database: airbyte_db
  user: airbyte_user
  existingSecret: production-hm-airbyte-secret
  existingSecretPasswordKey: POSTGRES_PASSWORD

However, after I deploy Airbyte Helm chart v0.199.0, I got error inside pod "airbyte-airbyte-bootloader":

Error: couldn't find key DATABASE_USER in Secret production-hm-airbyte/airbyte-airbyte-secrets

Also, I noticed Airbyte created a secret airbyte-airbyte-secrets during deployment with values

• DEFAULT_MINIO_ACCESS_KEY
• DEFAULT_MINIO_SECRET_KEY
• WORKLOAD_API_BEARER_TOKEN

And missing DATABASE_USER inside.

I try to append airbyte-bootloader section in my-values.yaml. Both

airbyte-bootloader:
  extraEnv:
    - name: DATABASE_USER
      value: airbyte_user

and

airbyte-bootloader:
  secrets:
    DATABASE_USER: airbyte_user

will give error

failed to create typed patch object (production-hm-airbyte/airbyte-airbyte-bootloader; /v1, Kind=Pod): .spec.containers[name="airbyte-bootloader-container"].env: duplicate entries for key [name="DATABASE_USER"]

I searched online and found

• https://github.com/airbytehq/airbyte/issues/22094
• https://github.com/airbytehq/airbyte/discussions/33835

But none of them provides a solution.

Any guide would be appreciate, thanks!

UPDATE (6/20/2024)

I found a clean solution! After I change from

global
  database:
    # ...
    secretName: production-hm-airbyte-secret
    user: airbyte_user
    passwordSecretKey: POSTGRES_PASSWORD
  # ...

to

global
  database:
    # ...
    secretName: production-hm-airbyte-secret
    userSecretKey: POSTGRES_USER_NAME
    passwordSecretKey: POSTGRES_PASSWORD
  # ...

It does not expect DATABASE_USER any more and can deploy smoothly!

P.S. I opened a ticket at https://github.com/airbytehq/airbyte/issues/39604 to hopefully user: airbyte_user way will work smoothly as well in future.

---

Orignal Answer

I found a workaround solution which is patching Kubernetes secret airbyte-airbyte-secrets to add missing DATABASE_USER immediately after deployment.

kubectl patch secret airbyte-airbyte-secrets \
  --namespace=production-hm-airbyte \
  --patch="{\"stringData\":{\"DATABASE_USER\":\"airbyte_user\"}}"

Then the pod airbyte-airbyte-bootloader can pick up this DATABASE_USER from Kubernetes secret airbyte-airbyte-secrets value.