Context: I'm building a high-throughput event processing application that needs to handle 100k events within 15 minutes. Currently, this is achieved using 200 parallel threads in IBM MQ within an IBM ACE application on OpenShift Red Hat CloudPak (app takes 9ms to process single event).
Challenge: We're migrating to Kafka and want to maintain the same throughput with 200 threads. However, when using Kafka node in IBM ACE message flow as far as i know , if I'm using same group id then I can connect only one thread to one single partition from same consumer group. Increasing partitions to 200 is undesirable due to (administrative overhead, cost).
How can we achieve 100k events/15 min with 200 parallel threads in Kafka without increasing the number of partitions?
Additional Information:
IBM ACE version (12.0.10.0)
IBM MQ 9.3.x
Duplicate message consumption should never occur
You cannot have more consumers than partitions in a group, but that doesn't mean those consumers are limited to single threads. See https://github.com/confluentinc/parallel-consumer