apache-kafkatransformationapache-kafka-connectdebeziumtombstone

Kafka Connect tombstone message doesn't apply InsertField transformation


InsertField transformation isn't applied in tombstone message.

I am using postgres 10.1 / debezium 1.2.1 version / kafka 2.4.1 version

for example.

{
...
    "transforms": "InsertField",
    "transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Key",
    "transforms.InsertField.static.field": "__host",
    "transforms.InsertField.static.value": "abc.com",
...
}
key - {"id":1, "__host": "abc.com"}, value - {"id":1, "__host": "abc.com", "col1": 1, "col2": "a"}
key - {"id":1, "__host": "abc.com"} value - {"id":1, "__host": "abc.com", "col1": null, "col2": null}
key - {"id":1} value - null

Why is it happened? Is it bug? or Am I wrong?


Solution

  • Expected behavior of InsertField transformation for tombstone messages:

    The key field is inserted, and the value remains null.

    But in your Kafka Connect version (2.4.1) there is a bug: the key field is not inserted. It was fixed in KAFKA-9707: InsertField.Key transformation should apply to tombstone records

    You can upgrade your Kafka Connect cluster to some fix version (e.g. 2.4.2).