sql-serverapache-kafkaapache-kafka-connectdebeziumapache-hudi

How to remove 'before' key from payload generated by debezium event for updates in SQL server


For every update in SQL server, debezium generates event payload with 'after' and 'before'. I want to get rid of 'before' without flattening the payload.

Use case:

I am using debezium with Kafka and hudi and have created a data lake, I don't need delete operations so I have skipped delete operations.

So I want to reduce the size of payload to half by eliminating the 'before' part so how we can achieve this?

TIA


Solution

  • Found a solution, used existing transforms. Following transformation works like a charm

    transforms=ReplaceField transforms.ReplaceField.type=org.apache.kafka.connect.transforms.ReplaceField$Value transforms.ReplaceField.exclude=before