apache-kafkacassandraapache-kafka-connectkafka-partition

How to specify the Kafka topic's partition in Kafka Connect Sink


I am trying to specify a topic partition for my Kafka Connect Sink. In particular, I am using the DataStax Apache Kafka Connector.

There is a good amount of documentation and resources related to specifying a topic partition for a Kafka Consumer, for example:

However, I haven't been able to find anything at all regarding how to specify what partition a given Kafka Connect Sink Connector reads from.

It seems like the Confluent connector developer docs imply that specifying partition should be possible, but I don't see any config that I can set in the generic Kafka Sink Configuration Properties docs nor in the DSE Kafka Connector configuration docs.

My understanding is that a Kafka Connect Sink is basically a specific implementation of a Kafka Consumer that writes to a given data store. If so, it should be possible to specify a partition, is that correct? Or am I misunderstanding something about how Kafka Connectors work?


Solution

  • You cannot specify partitions in the Connect API. It subscribes to all partitions, then distributes consumer instances amongst worker tasks as part of a consumer group.