[SOLVED] kafka + This server is not the leader for that topic-partition + kafka producer

kafka + This server is not the leader for that topic-partition + kafka producer

We are get many errors from server.log from each of 3 kafka machines ( we have 3 kafka in the cluster )

[2019-12-05 13:25:09,529] ERROR [ReplicaFetcherThread-0-1], Error for partition [jdty.dee.rules.time,91] to broker 1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2019-12-05 13:25:09,529] ERROR [ReplicaFetcherThread-0-1], Error for partition [jdty.dee.export.profiles,96] to broker 1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2019-12-05 13:25:09,529] ERROR [ReplicaFetcherThread-0-1], Error for partition [jdty.dee.control.tt.state,40] to broker 1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2019-12-05 13:25:09,529] ERROR [ReplicaFetcherThread-0-1], Error for partition [jdty.dee.control.tt,67] to broker 1001:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

from my understanding each topic is served by one or multiple Brokers - one is leader and the remaining brokers are followers.

A producer needs to send new messages to the leader Broker which internally replicate the data to all followers.

I assume, that your producer client does not connect to the correct Broker, its connect to a follower instead of the leader, and this follower rejects your send request.

so my question is - how to configure the producer configuration in order to avoid such these errors?

Solution

Follower brokers fetch from the leader broker (the leader is not pushing to followers). Hence, it seems that a follower broker tries to fetch from the wrong (leader) broker. This can happen if the leader of a partition changed. The corresponding follower broker should update its cluster metadata automatically to rediscover the new leader. If the error persists, it indicates that this follower broker has issues to update its metadata.