goapache-kafkasarama

Sarama Partition Consumer with ConsumerGroup


I have tried shopify/sarama library to consume kafka messages. I used both Consumer interface and ConsumerGroup interface. I can consume from specific partitions using ConsumePartition() method in Consumer. But when I use ConsumerGroup interface, I do not seem to have the capability to consume from a specific partition.

Is there a way for me to assign certain partitions to specific consumers inside a consumer group? Or is it something I cannot interfere with?


Solution

  • It seems like I cannot give exact partitions to consume when using ConsumerGroup. However I can choose the strategy that I want to assign partitions to my consumers, out of 2 strategy options.

    1. Balance Strategy Range

    BalanceStrategyRange is the default and assigns partitions as ranges to consumer group members. Example with one topic T with six partitions (0..5) and two members (M1, M2):

    M1: {T: [0, 1, 2]}
    M2: {T: [3, 4, 5]}
    
    1. Balance Strategy Round-Robin

    BalanceStrategyRoundRobin assigns partitions to members in alternating order. Example with topic T with six partitions (0..5) and two members (M1, M2):

    M1: {T: [0, 2, 4]}
    M2: {T: [1, 3, 5]}
    

    I can give this as a configuration when creating the ConsumerGroup.

    config.Consumer.Group.Rebalance.Strategy = BalanceStrategyRange