I am trying to specify a topic partition for my Kafka Connect Sink. In particular, I am using the DataStax Apache Kafka Connector.
There is a good amount of documentation and resources related to specifying a topic partition for a Kafka Consumer, for example:
However, I haven't been able to find anything at all regarding how to specify what partition a given Kafka Connect Sink Connector reads from.
It seems like the Confluent connector developer docs imply that specifying partition should be possible, but I don't see any config that I can set in the generic Kafka Sink Configuration Properties docs nor in the DSE Kafka Connector configuration docs.
My understanding is that a Kafka Connect Sink is basically a specific implementation of a Kafka Consumer that writes to a given data store. If so, it should be possible to specify a partition, is that correct? Or am I misunderstanding something about how Kafka Connectors work?
You cannot specify partitions in the Connect API. It subscribes to all partitions, then distributes consumer instances amongst worker tasks as part of a consumer group.