I am looking for a way to use producer with transaction, using fs2, however the current TransactionalProducer seems to be geared toward a scenario in which it is an end to end workflow, meaning consume-process-produce.
However, we would like to use it in a context where we are just producing message to kafka.
Is there a known way to achieve that with fs2-kafka ? I have tried to see how but it seems impossible, maybe i am missing something ?
After double checking, it is clear that the use case is not supported. I'm however curious as to why ? Is it for a specific reason, that i may need to be aware of while implementing my own solution, or is just that it is not done and won't never be, for no specific reason ? If someone could shed some light ?
Ultimately the only thing the transactional producer adds to enable.idempotence=true, acks=all is that the consumer offsets get committed as part of producing the message. Since the offsets being committed implies successful production and vice versa, this allows a consume-process-produce stream to process messages effectively-once (Confluent arguably stretches the exactly-once terminology a little bit), assuming everything in the process step is also idempotent.