[SOLVED] Debezium embedded connector with 'table.include.list' configured still captures other tables

Debezium embedded connector with 'table.include.list' configured still captures other tables

While trying to run a debezium engine with following configurations, I was expecting this engine instance only captures data change from tables(fge.record_demo) listed in 'table.include.list'.

Configuration config = Configuration.create()
    .with("name", "demo")
    .with("connector.class", OracleConnector.class.getName())
    .with("database.hostname", "*****")
    .with("database.port", "1521")
    .with("database.user", "*****")
    .with("database.password", "*****")
    .with("database.server.name", "ftpc-fge-connector")
    .with("database.dbname", "mesfa")
    .with("schema.include.list","ftpc_fge")
    .with("table.include.list", "ftpc_fge.record_demo")
    .with("log.mining.strategy", "online_catalog")
    .with("decimal.handling.mode","string")
    .with("topic.prefix","ftpc-fge-cdc")
    .with("offset.storage", FileOffsetBackingStore.class.getName())
    .with("offset.storage.file.filename","D:\\workspace\\site-portal\\extractor\\debezium-offset\\offset.dat")
    .with("database.history", "io.debezium.relational.history.FileDatabaseHistory")
    .with("database.history.file.filename", "D:\\workspace\\site-portal\\extractor\\debezium-offset\\history.dat")
    .with("schema.history.internal", "io.debezium.storage.file.history.FileSchemaHistory")
    .with("schema.history.internal.file.filename", "D:\\workspace\\site-portal\\extractor\\debezium-offset\\schemahistory.dat")
    .with("offset.flush.interval.ms","60000")
    .build();

Surprisingly I noticed the info consoled as below and so on:

2024-08-07 18:58:37 [debezium-oracleconnector-ftpc-fge-cdc-change-event-source-coordinator] INFO i.d.r.RelationalSnapshotChangeEventSource - Snapshot step 5 - Reading structure of captured tables 2024-08-07 18:58:37 [debezium-oracleconnector-ftpc-fge-cdc-change-event-source-coordinator] INFO i.d.c.o.OracleSnapshotChangeEventSource - All eligible tables schema should be captured, capturing: [ MESFA.FTPC_ADAYO.AA_FORD_GAP_CHECKLIST, MESFA.FTPC_ADAYO.AA_FORD_GLUE_CHECKLIST, MESFA.FTPC_ADAYO.AA_FORD_MATURINGRECORD, MESFA.FTPC_ADAYO.AA_FORD_SCALE_MAIN, MESFA.FTPC_ADAYO.AA_FORD_SCREW_CHECKLIST, MESFA.FTPC_ADAYO.AA_FOTON_AUTH_CODE, MESFA.FTPC_ADAYO.AA_INCOMMING_MATERIAL, MESFA.FTPC_ADAYO.AA_INV_ASSIST_MATERIAL_TYPE,...

After that it started to capture all tables from all schemas, instead of capturing ftpc_fge.record_demo only.

The version of the debezium connector I'm using is 2.7.0.Final

implementation 'io.debezium:debezium-embedded:2.7.0.Final'

implementation 'io.debezium:debezium-connector-oracle:2.7.0.Final'

And the oracle version is 11g

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

I tried to delete the offset data then reboot this application, the problem still remained.

After referencing this question I changed the 'table.include.list' config to 'table.whitelist', still not working.

Solution

After adding this configuration, it finally worked out properly.

.with("schema.history.internal.store.only.captured.tables.ddl",true)

The point is that debezium engine appears to performs snapshots over whole database, before which it won't perform any capturing over designated tables.

By enabling the initial snapshot to capture schema data for tables that are not part of the original capture set, Debezium prepares the connector to readily capture event data from these tables should that later become necessary.

According to the official document, we could use the "store.only.captured.tables.ddl" configuration to prevent the whole database snapshot.