I'd like to parallelize the write to kafka, that is having multiple producer sending data to kafka, although from within an akka-stream. In other my stream would have several initial stage from the source, and then when arriving at sending the data, i would like to have about 16 worker sending the data at the same time.
I wonder if i need to embed Akka Streams Kafka in an akka-stream Graph DSL and use a balancer for that, or if there is an easier solution. Also, simply, if someone has done anything like that in general that would be great.
Akka Streams Kafka (aka Reactive Kafka) has got a specific setting for the producer parallelism (see the docs)
akka.kafka.producer {
# Tuning parameter of how many sends that can run in parallel.
parallelism = 100
...
}
Have you tried tuning it to solve your problem?