I am trying to setup a kafka statefulset on Kubernetes scaled up to 3 pods & want to have all kafka brokers to be able to connect with SASL_PLAINTEXT mechanism, however I'm getting the result that only one broker is able to connect & it blocks the rest of brokers. Anyone could please help me make them all able to connect with SASL_PLAINTEXT: here is the deployment manifest:
java.lang.IllegalArgumentException: requirement failed: Configured end points xx.x.x.xxx:9094 in advertised listeners are already registered by broker 2
here is the mounted kafka_jaas.json:
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka-user"
password="kafka-password"
user_kafka="kafka-password";
};
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="kafka-user"
password="kafka-password"
user_kafka="kafka-password";
};
and the listeners config:
- name: KAFKA_LISTENERS
value: "INSIDE://:9092,LB://:9094,CONTROLLER://:9093"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INSIDE://:9092,LB://xx.x.x.xxx:9094,CONTROLLER://:9093"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INSIDE:SASL_PLAINTEXT,LB:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INSIDE"
- name: KAFKA_CONTROLLER_LISTENER_NAME
value: "CONTROLLER"
- name: KAFKA_CONTROLLER_SHUTDOWN_ENABLE
value: "false"
- name: KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL
value: "PLAIN"
- name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
value: "PLAIN"
- name: KAFKA_SASL_ENABLED_MECHANISMS
value: "PLAIN"
(xx.x.x.xxx is replaced by the external IP address of a load balancer for the kafka service) & the Kafka image is: wurstmeister/kafka:2.12-2.2.1 zookeeper image: zookeeper:3.5
I resolved it by pointing the listners on podIP of replicasets
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: KAFKA_LISTENERS
value: "INSIDE://:9092"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INSIDE://$(MY_POD_IP):9092"