spring-integration

Balance consumers with KinesisMessageDrivenChannelAdapter


How does KinesisMessageDrivenChannelAdapter currently handle balancing between multiple instances? I would like that the shards are distributed somewhat evenly when I spin up another instance for scaling.

I found this older ticket https://github.com/spring-projects/spring-integration-aws/issues/99. It seems like there is no rebalancing.

Thanks for your input!


Solution

  • There is no consumer group management in AWS Kinesis and that is not easy to implement from this KinesisMessageDrivenChannelAdapter client. We would need some other independent application to coordinate that behavior. That is already out of spring-integration-aws scope.

    Currently it works like: first in - first win. So, indeed might be a situation when all your shards are consumed by a single instance.