hadoopapache-kafkaapache-kafka-connectconfluent-platform

Need to move small JSON messages from Kafka to HDFS with Kafka Connect but without using Confluent libs, if not completely free

I'd like to use Kafka Connect to move JSON messages from Kafka to HDFS and then Impala, only using OpenSource libs.

I was trying to understand if I can use the Confluent Sink library for Kafka Connect, without the need to use the entire Confluent distribution.

Are there other and/or better options to achieve this?

Solution

  • The Kafka Connect HDFS 2 Sink is available under the Confluent Community Licence. It is a plugin for Apache Kafka; you do not have to run Confluent Platform to use it.