Strimzi does not parse secret when deploying Kafka Connector with Helm

I am struggling with io.strimzi.kafka.KubernetesSecretConfigProvider when deploying connector to my Kafka Connect service in k8s. Pod log returns error when trying to create connector:

ERROR Uncaught exception in REST call to /connectors/pg-source-connector/config (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper)
org.apache.kafka.common.config.ConfigException:
Invalid path . It has to be in format <namespace>/<secret> (or <secret> for default namespace).
at io.strimzi.kafka.KubernetesResourceIdentifier.fromConfigString(KubernetesResourceIdentifier.java:33)

Seems that for some reason my connector can't parse "${...}" values in it's template. I'm not sure if the problem is with Strimzi or some Kubernetes settings. I also previously had problem with parsing when I tried to use io.strimzi.kafka.KubernetesEnvProvider

Here are my templates. If necessary, I'll provide Role and Role-binding, but I doubt the problem lies with them.

kafka-connect.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kc-cluster
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: {{ .Values.repository.image }}:{{ .Values.repository.tag }}
  template:
    pod:
      securityContext:
        runAsUser: 1
        runAsNonRoot: true
    connectContainer:
      securityContext:
        capabilities:
          drop:
            - ALL
        privileged: false
        runAsUser: 1
        runAsGroup: 1
        runAsNonRoot: true
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        seccompProfile:
          type: RuntimeDefault
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      memory: 1000Mi
    requests:
      cpu: 500m
      memory: 500Mi
  bootstrapServers: {{ .Values.kafka.bootstrapServers }}
  tls:
    trustedCertificates:
      - secretName: kc-ca-cert
        certificate: ca.crt
  authentication:
    type: scram-sha-512
    username: {{ .Values.kafka.username }}
    passwordSecret:
      secretName: {{ .Values.kafka.username }}
      password: password
  config:
    config.providers: secrets
    config.providers.secrets.class: io.strimzi.kafka.KubernetesSecretConfigProvider
    ssl.cipher.suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    ssl.protocol: TLSv1.3
    group.id: kc
    offset.storage.topic: kc-offsets
    config.storage.topic: kc-configs
    status.storage.topic: kc-statuses
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
    metrics.reporter: "none"
    log4j.logger.org.apache.kafka.connect.runtime: DEBUG
    log4j.logger.org.apache.kafka.connect.connector: DEBUG

secret.yaml:

kind: Secret
apiVersion: v1
metadata:
  name: db-creds
  namespace: kc-namespace
  annotations:
    kubesphere.io/creator: k8suser
data:
  password: EnCrYpTeDpAsSwOrD123
  username: EnCrYpTeDuSeRnAmE456
type: Opaque

pg-source-connector.yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: pg-source-connector
  namespace: kc-namespace
  labels:
    strimzi.io/cluster: kc-cluster
spec:
  class: io.debezium.connector.postgresql.PostgresConnector
  tasksMax: 1
  config:
    database.dbname: "database"
    database.hostname: "12.345.678.910"
    database.password: "${secrets:kc-namespace/db-creds/password}"
    database.port: "5432"
    database.user: "${secrets:kc-namespace/db-creds/username}"
    key.converter: "org.apache.kafka.connect.json.JsonConverter"
    key.converter.schemas.enable: "false"
    plugin.name: "pgoutput"
    primary.key: "id"
    publication.autocreate.mode: "disabled"
    publication.name: "test_kc"
    schema.include.list: "public"
    snapshot.include.collection.list: "public.test_kc"
    snapshot.mode: "initial"
    table.include.list: "public.test_kc"
    time.precision.mode: "connect"
    topic.creation.default.cleanup.policy: "delete"
    topic.creation.default.partitions: "3"
    topic.creation.default.replication.factor: "2"
    topic.delimiter: "_"
    topic.prefix: "kc-pg"
    value.converter: "org.apache.kafka.connect.json.JsonConverter"
    value.converter.schemas.enable: "false"

When I check deployed Kafka Connect in Kubernetes - there are no /opt/kafka/secrets directory, where creds should be stored. I understand that the problem lies in "Invalid path .", but can't figure out the cause of it.

There are two things to ensure:

1. That the special characters such as $, {, or } are not removed by your Helm Chart

2. That you use the format for the config provider correctly

I cannot comment much on the first point as it is your Helm chart and I have no idea what it does. But based on the error it probably works fine as it seems to have failed only later.

For the second part, you seem to have it wrong because the structure is not <namespace>/<secretName>/<key> as you see, to use it. It is <namespace>/<secretName>:<key> . So you will need to fix it. E.g.:

database.password: "${secrets:kc-namespace/db-creds:password}"

You can see that also in the configuration provider docs: https://github.com/strimzi/kafka-kubernetes-config-provider?tab=readme-ov-file#using-the-configuration-provider