Let's suppose a Kafka streams that operates stateful has an input topic with 16 partitions and the number of instances (or tasks) is also 16. I understand that the default partitioner of the Kafka producer creates and distributes hash values as many as the number of partitions. Therefore, the state store (Rocksdb), which is also in the same task, is likely to experience a different distribution of hash values as the number of partitions increases.
For example, if we increase the number of partitions from 16 to 32, I think the state store's key distribution, which was previously distributed within the 16 partitions, would also be affected because now it's distributed within 32 partitions.
I expect some sort of rebalancing to occur behind the scenes, but despite my extensive search, I can only find content related to fault tolerance.
If my simple thought is correct, I'm wondering if I need to recreate the streams application from scratch if I want to increase the number of partitions. Could anyone shed light on how the partition increase affects the key distribution in the state store and whether I really need to rebuild my application? Thank you in advance.
The number of partitions is a fixed upper limit for scaling up your Streams application. Therefore, it's wise to plan ahead and start with more partitions than you have instances, so that there is room to scale up later. Having a lot more partitions than instances has some cost, but will be handled fine by Kafka Streams.
If you already have a Streams deployment that you want to scale up beyond the current number of partitions, you indeed need to recreate your the state from scratch. One option is to shut down the Streams application, wipe the state directory and repartition all input, output and internal topics (repartition topics and changelog topics). Then start the application. It will restore the state from the changelog topic, and thereby apply the new partitioning in the local state as well. Alternatively, you can recreate the Streams application completely (internal + output topics) and hotswap downstream consumers to the new output topics - but it will be slower.
There is an accepted, but not yet implemented KIP that will (partially) improve this situation: