[SOLVED] Why are writes to kafka slow if I add a new broker?

Why are writes to kafka slow if I add a new broker?

I have a kafka cluster and I want to rotate 1 broker.

process:

I'm first adding new brokers
I "assign" partitions to be replicated by these extra brokers (I'm just ADDING replication, see terminal below)
Once they are in the ISR I will remove the broker that I want to rotate from the partitions assignment
upgrade my broker
add it back to the partitions
finally remove the extra brokers that I added in the first place

So I'm just at step 1, I'm ADDING extra brokers. I want to rotate broker 6 and I added 7 to partition 1 (where broker 6 is assigned), see below.

Topic: mytopic  PartitionCount: 1024    ReplicationFactor: 4    Configs: min.insync.replicas=2,retention.ms=259200000,segment.ms=604800000
    Topic: mytopic  Partition: 0    Leader: 1   Replicas: 1,2,3 Isr: 1,2,3
    Topic: mytopic  Partition: 1    Leader: 2   Replicas: 2,1,6,7   Isr: 1,2,6

I have server.properties: min.insync.replicas=2 and when I write, I do acks=all

Problem:

As I'm waiting for 7 to be in the ISR, I noticed that my writes to kafka are much slower (800ms vs 200ms) What can cause that?

Solution

Well, you have thousands of partitions, and potentially many topics. It is unclear how large each of these partitions actually is, but if you have many GB of data, then yes, latency is expected.

Specifically, replication is causing high network and disk IO on the brokers, so latency may increase while this finishes... You could address this with unique volume mounts (separate physical disks) set to log.dirs setting, and physical network links between the brokers, separate from external clients.

You also have acks=all, which also relies on replication to finish. You can use Quotas to limit replication speed.

Also, kafka-reassign-partitions script for should be used. Increasing topic replication factor alone doesn't make topics use new brokers. https://kafka.apache.org/documentation/#basic_ops_cluster_expansion