I am trying to gradually enable ACLs on a existing cluster (3.1.0 bitnami helm chart) which is configured like this :
listeners=INTERNAL://:9093,CLIENT://:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT
advertised.listeners=CLIENT://$(MY_POD_NAME)-k8s.dev.host.com:4430,INTERNAL://$(MY_POD_NAME).message-broker-dev-kafka-headless.message-broker-dev.svc.cluster.local:9093
The kafka-k8s.dev.host.com:4430 is internally forwarded to the CLIENT listener on 9092 For now, we are doing TLS termination on the LB, hence the PLAINTEXT on the CLIENT listener but using SSL security.protocol :
kafkacat -b kafka-k8s.dev.host.com:4430 -X security.protocol=SSL -L
The plan is to add 2 new listeners that will require SASL auth, migrate the clients to the listeners & deprecate the existing listeners. The new configuration will look like this :
listeners=INTERNAL://:9093,CLIENT://:9092,SASL_INTERNAL://:9095,SASL_CLIENT://:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,SASL_INTERNAL:SASL_PLAINTEXT,SASL_CLIENT:SASL_PLAINTEXT
advertised.listeners=CLIENT://$(MY_POD_NAME)-k8s.dev.host.com:4430,INTERNAL://$(MY_POD_NAME).message-broker-dev-kafka-headless.message-broker-dev.svc.cluster.local:9093,SASL_CLIENT://$(MY_POD_NAME)-sasl-k8s.dev.host.com:4430,SASL_INTERNAL://$(MY_POD_NAME).message-broker-dev-kafka-headless.message-broker-dev.svc.cluster.local:9095
allow.everyone.if.no.acl.found=true
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=PLAIN
After creating some SCRAM-SHA-512 users and applying ACLs to existing topics, everything is working fine on the SASL_INTERNAL listener but not on the SASL_CLIENT :
$ kafkacat -b message-broker-dev-kafka-headless.message-broker-dev:9095 -C -t protected-topic-v1 -X security.protocol=SASL_PLAINTEXT -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=demo-user -X sasl.password=secret
{"userId":"1225"}
% Reached end of topic protected-topic-v1 [0] at offset 22
$ kafkacat -b kafka-sasl-k8s.dev.host.com:4430 -C -t protected-topic-v1 -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=demo-user -X sasl.password=secret
%3|1669825033.516|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://kafka-sasl-k8s.dev.host.com:4430/bootstrap]: sasl_ssl://kafka-sasl-k8s.dev.host.com:4430/bootstrap: SASL SCRAM-SHA-512 mechanism handshake failed: Broker: Request not valid in current SASL state: broker's supported mechanisms: (after 44ms in state AUTH_HANDSHAKE)
The kafka-sasl-k8s.dev.host.com:4430 is internally forwarded to the SASL_CLIENT listener on 9094 (and again using TLS termination on LB, so SASL_SSL instead of SASL_PLAINTEXT) For now, I'm not totally sure if I missed a kafka configuration or messed a network configuration.
Thanks in advance.
Auto-answering, it was a network issue.
kafka-sasl-k8s.dev.host.com:4430 was sending traffic to 9092 & not 9094 as expeccted