[SOLVED] Connecting Docker image for debezium/connect:1.7 with a Kafka cluster that has SSL/SASL security

Connecting Docker image for debezium/connect:1.7 with a Kafka cluster that has SSL/SASL security

Basically the title is my struggle. I've seen a lot of examples where people were in a "similar" situation... But my problem here is that I can't even deploy the image and make the REST endpoint available, since the image itself cannot connect with Kafka.

I couldn't find anywhere in the docs, how to supply API_KEY and API_SECRET in the docker image, so the debezium/connect:1.7 image can connect with Kafka... My docker-compose looks like this:

wmv-debezium-connect:
    container_name: wmv-debezium-connect
    build:
      context: .
      dockerfile: Dockerfile
    env_file:
      - .env
    restart: always
    # depends_on:
    #   kafka:
    #     condition: service_healthy
    #   mssql:
    #     condition: service_healthy
    ports:
      - ${REST_PORT:-8083}:${REST_PORT:-8083}
    expose:
      - "${REST_PORT:-8083}"
    healthcheck:
      test:
        [
          "CMD",
          "curl",
          "--silent",
          "--fail",
          "-X",
          "GET",
          "http://${DEBEZIUM_CONNECT_HOST:-wmv-debezium-connect}:${REST_PORT:-8083}/connectors",
        ]
      start_period: 10s
      interval: 10s
      timeout: 10s
      retries: 20

P.S.: depends_on is commented out since I'm trying to use both database and kafka in my deployed clusters.

And I'm supplying the env vars through .env file, which looks like:

BOOTSTRAP_SERVERS=my_kafka_cluster_url:9092

# this is what I tried, but no success... Also tried without `CONNECT_` prefix
CONNECT_KAFKA_SECURITY_PROTOCOL=SASL_SSL
CONNECT_KAFKA_SASL_MECHANISM=PLAIN
CONNECT_KAFKA_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="my_api_key" password="my_api_secret";'

CONFIG_STORAGE_TOPIC=dbz-config
OFFSET_STORAGE_TOPIC=dbz-offset
STATUS_STORAGE_TOPIC=dbz-status
DB_KAFKA_HISTORY_TOPIC=data-changes
GROUP_ID=debezium-connect
KAFKA_API_KEY=my_api_key
KAFKA_API_SECRET=my_api_secret
REST_PORT=8083
DEBEZIUM_CONNECT_HOST=wmv-debezium-connect
DEBEZIUM_CONNECTOR_NAME=db-connector

# Database vars for connector (to use in the REST API)
DB_HOST_NAME=my_db_host
DB_SERVER_NAME=wmv
DB_PASSWORD=my_db_password
DB_USER=us
DB_PORT=1433
DB_NAME=MyDBName

Does anyone ever experienced this and know how to supply informations of SSL/SASL while deploying this image? When the image is deployed, I follow the logs and I see that the kafka config that the image inittialize isn't correct(because is missing these security data)

2023-08-24 07:28:02,965 INFO   ||  AdminClientConfig values: 
bootstrap.servers = [my_kafka_cluster_url:9092]
client.dns.lookup = use_all_dns_ips
client.id = 
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS

I'm kinda going crazy on this, hard to find a solution 😂

Appreciate any help!!

Solution

For those stuck in the same as I: You basically can replace any internal docker variable some.variable to docker as CONNECT_SOME_VARIABLE.

So this was the solution!