node.jsapache-kafkakafkajs

Kafka broker error: 'There is no leader for this topic-partition as we are in the middle of a leadership election.'


Im running a Kafka Broker Cluster containing 3 brokers. I connected my Nodejs application with cluster. When all three brokers are up, application is connected. everything works fine. but when i shutdown the leader broker in the cluster. then application retries to connect with broker in cluster. but i continously get the error, 'There is no leader for this topic-partition as we are in the middle of a leadership election.'

That is my producer application code.


console.log("producer..........")
const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:8092', 'localhost:8093', 'localhost:8094']
})

const producer = kafka.producer()
// const consumer = kafka.consumer({ groupId: 'test-group' })

const run = async () => {
  // Producing
  await producer.connect()
  setInterval(async ()=> {
    await producer.send({
      topic: 'test-topic',
      messages: [
        { value: 'Hello KafkaJS user!' },
      ],
    })

  }, 2000)
}

run().catch(console.error)

and im running brokers on docker. that is my docker-compose file.

version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    container_name: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    networks:
      - kafka-network

  kafka1:
    image: confluentinc/cp-kafka
    container_name: kafka1
    ports:
      - 8092:8092
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,CONNECTIONS_FROM_HOST://localhost:8092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
    volumes:
      - kafka1-data:/var/lib/kafka/data
    depends_on:
      - zookeeper
    networks:
      - kafka-network  

  kafka2:
    image: confluentinc/cp-kafka
    container_name: kafka2
    ports:
      - 8093:8093
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093,CONNECTIONS_FROM_HOST://localhost:8093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
    volumes:
      - kafka2-data:/var/lib/kafka/data
    depends_on:
      - zookeeper
    networks:
      - kafka-network

  kafka3:
    image: confluentinc/cp-kafka
    container_name: kafka3
    ports:
      - 8094:8094
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094,CONNECTIONS_FROM_HOST://localhost:8094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
    volumes:
      - kafka3-data:/var/lib/kafka/data
    depends_on:
      - zookeeper
    networks:
      - kafka-network

volumes:
  kafka1-data:
  kafka2-data:
  kafka3-data:
networks:
  kafka-network:

Solution

  • You've not shown how you create any specific topic, so I'll assume you are letting them get automatically created, and by default this will have only one partition, and one replica.

    If you shut down the broker hosting either, as the error says - it is unavailable to find the singular leader for that topic partition.