apache-kafkaflume

How to send key, value messages with flume to a kafka producer


In console you add producer properties --property "parse.key=true" --property "key.separator=:" to produce key-value data into Kafka, but how to do this with flume? I tried to add

a1.sinks.k1.producer.parse.key=true 
a1.sinks.k1.producer.key.separator=:

in .conf file but was of no avail, the kafka treated the key like a string.


Solution

  • Those are console-producer CLI arguments, not ProducerConfig properties for Kafka (which are passed to Flume)

    The key will always be a string, but you pass it via the headers of the Flume record

    https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L193