springspring-kafkaspring-retrykafka-topic

Correct Number of Partitions/Replicas for @RetryableTopic Retry Topics


Hello Stack Overflow community and anyone familiar with spring-kafka!

I am currently working on a project which leverages the @RetryableTopic feature from spring-kafka in order to reattempt the delivery of failed messages. The listener annotated with @RetryableTopic is consuming from a topic that has 50 partitions and 3 replicas. When the app is receiving a lot of traffic, it could possibly be autoscaled up to 50 instances of the app (consumers) grabbing from those partitions. I read in the spring-kafka documentation that by default, the retry topics that @RetryableTopic autocreates are created with one partition and one replica, but you can change these values with autoCreateTopicsWith() in the configuration. From this, I have a few questions:


Solution

  • The retry topics should have at least as many partitions as the original (by default, records are sent to the same partition); otherwise you have to customize the destination resolution to avoid the warning log. See Destination resolver returned non-existent partition

    50 partitions might be overkill unless you get a lot of retried records.

    It's up to you how many replicas you want, but in general, yes, I would use the same number of replicas as the original.

    Only you can decide what are the "correct" numbers.