kubernetesapache-kafkaapache-zookeeperjaassasl

Configure SASL_PAINTEXT over multiple Kafka Brokers


I am trying to setup a kafka statefulset on Kubernetes scaled up to 3 pods & want to have all kafka brokers to be able to connect with SASL_PLAINTEXT mechanism, however I'm getting the result that only one broker is able to connect & it blocks the rest of brokers. Anyone could please help me make them all able to connect with SASL_PLAINTEXT: here is the deployment manifest:

java.lang.IllegalArgumentException: requirement failed: Configured end points xx.x.x.xxx:9094 in advertised listeners are already registered by broker 2

here is the mounted kafka_jaas.json:

KafkaClient {
    org.apache.kafka.common.security.plain.PlainLoginModule required 
    username="kafka-user" 
    password="kafka-password" 
    user_kafka="kafka-password";
    };
KafkaServer {
    org.apache.kafka.common.security.plain.PlainLoginModule required 
    username="kafka-user" 
    password="kafka-password" 
    user_kafka="kafka-password";
    };

and the listeners config:

    - name: KAFKA_LISTENERS
      value: "INSIDE://:9092,LB://:9094,CONTROLLER://:9093"
    - name: KAFKA_ADVERTISED_LISTENERS
      value: "INSIDE://:9092,LB://xx.x.x.xxx:9094,CONTROLLER://:9093"
    - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
      value: "INSIDE:SASL_PLAINTEXT,LB:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT"

    - name: KAFKA_INTER_BROKER_LISTENER_NAME
      value: "INSIDE"

    - name: KAFKA_CONTROLLER_LISTENER_NAME
      value: "CONTROLLER"

    - name: KAFKA_CONTROLLER_SHUTDOWN_ENABLE
      value: "false"
    - name: KAFKA_SASL_MECHANISM_CONTROLLER_PROTOCOL
      value: "PLAIN" 
    - name: KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL
      value: "PLAIN" 
    - name: KAFKA_SASL_ENABLED_MECHANISMS
      value: "PLAIN" 

(xx.x.x.xxx is replaced by the external IP address of a load balancer for the kafka service) & the Kafka image is: wurstmeister/kafka:2.12-2.2.1 zookeeper image: zookeeper:3.5


Solution

  • I resolved it by pointing the listners on podIP of replicasets

        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
            - name: KAFKA_LISTENERS
              value: "INSIDE://:9092"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: "INSIDE://$(MY_POD_IP):9092"