I have an application with the need to pass messages into multiple layers of processing. I need to do this because all the new messages should be put into the first generic topic so they can be processed to calculate a type and after that, they should be put into the other topic (for further processing), and from now on all the messages with the same key, go directly to the second topic automatically.
I'm planning to create multiple topics for each layer. Messages first go into the first layer and get processed, and then they should be sent to the next layer (another topic) and this might happen again for the next layer. I was wondering what is the best practice for this. Is it ok to produce messages in the consumer? Or is there any other better solution for this?
Producing within a consumer is perfectly acceptable. Python libraries such as Faust make this much simpler.