apache-kafkakafka-topickafka-partition

How Kafka guarantee the messages order while we increase the partitions in runtime?


I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

If so, what if all messages are provided with keys, then none of them will be mapped to new partition? This doesn't make sense to me.

If not, then how Kafka guarantee the order to messages with the same key in order?


Solution

  • I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

    No, the new messages will be partitioned based on the new number of partitions. Old messages will not get re-partitioned.

    If not, then how Kafka guarantee the order to messages with the same key in order?

    There are no guarantees when changing the number of partitions. When increasing the number of partitions at runtime the ordering can, or better, will change.

    Changing a partition of a topic that already includes data is therefore not recommended if you are relying on ordering of your data. What you could do is

    Remember, it is not possible to decrease the number of partitions.