I am setting up a Confluent Kafka Cluster (Community) with 3 Zookeper and 5 Kafka Broker nodes . The requirement is that we should be able to continue in live environment even if 2 broker nodes are down.
What should be the recommended -replication factor , -in sync replica for topics with 50 partitions.
In most case the suggested replication factor is 3 . What would be the impact if we increase that to 5 in the mentioned cluster configuration
Setting the replication factor to 5 would mean that all partitions exist on all brokers in the cluster. If two brokers are down, then the replication factor requirement is no longer met and your topics will be under-replicated (should give a warning).
min.insync.replicas should then be set to 3 (or less), otherwise producing a message with acks = all would fail. Producing a message with acks set to 1, 2, or 3 would also work on higher values of min.insync.replicas.
Also note that while two nodes are down, you can't create new topics with a replication factor of 5 (also see KIP-409).