It is known that Kafka can provide exactly-once delivery semantic after 0.11.0.0. And I expect to make filebeat output to Kafka that way. According to KIP 98, it requires some configs to be set properly, including enable.idempotence, transaction.timeout.ms and transactional.id for producer (i.e., filebeat) . But I cannot find these configs even in the latest version of filebeat. So, I want to know if it is supported and how to achieve that if it is a yes.
Currently, from the documentation of Filebeat and resources surrounding it, it is not possible to have exactly-once delivery with Filebeat. It currently utilizes the at-least once delivery, meaning you'll get the message 1 or more times.
Although Kafka can support exactly-once semantics, it appears that Filebeat does not support exactly-once semantics. Filebeat would need to implement those configurations to support exactly-once delivery.