I am running Debezium with column.include.list configured to a subset of columns on each of the observed tables of the source MySQL database. Changes to records on the source tables are being successfully published to Kafka, with each message's values before & after only containing that subset of columns.
However, Debezium still publishes messages when changes occur on columns of the observed tables that are not in column.include.list. Those events are unnecessary to my downstream consumers, so I'd like to prevent them from being produced.
I only want changes to columns found in column.include.list to produce messages to Kafka. What is the preferred way to do this?
Using SMT Filtering seems like one way to do it—a filter that compares the before & after values and filters out any messages in which there is no difference. Is there a simpler way? Maybe a config for this behavior I missed in my search?
Update:
This should be doable now that my PR is merged (upon the next release).
https://github.com/debezium/debezium/pull/4378
(From our discussion in the comments)
There's an open issue describing exactly what you expect, but not yet been picked up for development.
https://issues.redhat.com/browse/DBZ-2979
Thus it seems currently you need to rely on SMTs to filter events/messages not related to monitored columns.