amazon-s3sink

S3 Sink Connector not creating key or header files within bucket


I'm using the latest version of the S3 Sink Connector (v10.0.5) and enabled both the kafka.keys and kafka.headers but only value files are being created. Here's a copy of our config:

connector.class=io.confluent.connect.s3.S3SinkConnector
format.class=io.confluent.connect.s3.format.avro.AvroFormat
errors.log.include.messages=true
s3.region=eu-west-1
flush.size=1
tasks.max=1
errors.log.enable=true
s3.bucket.name=sink-connector-backup-bcs
schema.compatibility=FULL
topics=onboardingStatus

store.kafka.keys=true
store.kafka.headers=true
keys.format.class=io.confluent.connect.s3.format.avro.AvroFormat
headers.format.class=io.confluent.connect.s3.format.avro.AvroFormat

value.converter=io.confluent.connect.avro.AvroConverter
value.converter.enhanced.avro.schema.support=true
value.converter.schema.registry.url=http://.......eu-west-1.compute.internal:8083

key.converter=io.confluent.connect.avro.AvroConverter
key.converter.enhanced.avro.schema.support=true
key.converter.schema.registry.url=http://.......eu-west-1.compute.internal:8083

names=s3-sink-onboardingStatus
s3.sse.kms.key.id=arn:aws:kms:eu-west-1:.......:key/......
partitioner.class=io.confluent.connect.storage.partitioner.DefaultPartitioner
storage.class=io.confluent.connect.s3.storage.S3Storage
auto.offset.reset=earliest

what am I missing?


Solution

  • After some digging on our instance of AWS I discovered that we werent actually using the latest version of the S3 Sink Connector. Updated to the latest version and it worked. Did notice a potential bug: if the header or key for a message is empty (and you attempt to output that file type) then the sink connector fails