Im running a Kafka Broker Cluster containing 3 brokers. I connected my Nodejs application with cluster. When all three brokers are up, application is connected. everything works fine. but when i shutdown the leader broker in the cluster. then application retries to connect with broker in cluster. but i continously get the error, 'There is no leader for this topic-partition as we are in the middle of a leadership election.'
That is my producer application code.
console.log("producer..........")
const { Kafka } = require('kafkajs')
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['localhost:8092', 'localhost:8093', 'localhost:8094']
})
const producer = kafka.producer()
// const consumer = kafka.consumer({ groupId: 'test-group' })
const run = async () => {
// Producing
await producer.connect()
setInterval(async ()=> {
await producer.send({
topic: 'test-topic',
messages: [
{ value: 'Hello KafkaJS user!' },
],
})
}, 2000)
}
run().catch(console.error)
and im running brokers on docker. that is my docker-compose file.
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper
container_name: zookeeper
ports:
- 2181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
networks:
- kafka-network
kafka1:
image: confluentinc/cp-kafka
container_name: kafka1
ports:
- 8092:8092
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,CONNECTIONS_FROM_HOST://localhost:8092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
volumes:
- kafka1-data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- kafka-network
kafka2:
image: confluentinc/cp-kafka
container_name: kafka2
ports:
- 8093:8093
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9093,CONNECTIONS_FROM_HOST://localhost:8093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
volumes:
- kafka2-data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- kafka-network
kafka3:
image: confluentinc/cp-kafka
container_name: kafka3
ports:
- 8094:8094
environment:
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9094,CONNECTIONS_FROM_HOST://localhost:8094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONNECTIONS_FROM_HOST:PLAINTEXT
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE: "false"
volumes:
- kafka3-data:/var/lib/kafka/data
depends_on:
- zookeeper
networks:
- kafka-network
volumes:
kafka1-data:
kafka2-data:
kafka3-data:
networks:
kafka-network:
You've not shown how you create any specific topic, so I'll assume you are letting them get automatically created, and by default this will have only one partition, and one replica.
If you shut down the broker hosting either, as the error says - it is unavailable to find the singular leader for that topic partition.