apache-kafkaflumeflume-ng

how to send key value to kafka via Flume

I can send messages to kafka via Flume,but how to send key value messages to kafka ?

such as I use tailf log file as source,how can I send key value messages?

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command =tail -F /apps/logs/mysql_status.log
Solution

  • You'll have to use Flume headers

    Kafka Sink uses the topic and key properties from the FlumeEvent headers to send events to Kafka. If topic exists in the headers, the event will be sent to that specific topic, overriding the topic configured for the Sink. If key exists in the headers, the key will used by Kafka to partition the data between the topic partitions

    Headers can be added with interceptors

    .http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html#flume-interceptors