I have tried shopify/sarama library to consume kafka messages. I used both Consumer
interface and ConsumerGroup
interface. I can consume from specific partitions using ConsumePartition()
method in Consumer
. But when I use ConsumerGroup
interface, I do not seem to have the capability to consume from a specific partition.
Is there a way for me to assign certain partitions to specific consumers inside a consumer group? Or is it something I cannot interfere with?
It seems like I cannot give exact partitions to consume when using ConsumerGroup
. However I can choose the strategy that I want to assign partitions to my consumers, out of 2 strategy options.
BalanceStrategyRange is the default and assigns partitions as ranges to consumer group members. Example with one topic T with six partitions (0..5) and two members (M1, M2):
M1: {T: [0, 1, 2]}
M2: {T: [3, 4, 5]}
BalanceStrategyRoundRobin assigns partitions to members in alternating order. Example with topic T with six partitions (0..5) and two members (M1, M2):
M1: {T: [0, 2, 4]}
M2: {T: [1, 3, 5]}
I can give this as a configuration when creating the ConsumerGroup.
config.Consumer.Group.Rebalance.Strategy = BalanceStrategyRange