sql-serverapache-kafkasnapshotdebeziumadhoc

Ad-Hoc Snapshot with Debezium SQL Server Source Connector using Kafka Signal doesn't work

I've following the documentation at https://debezium.io/documentation/reference/2.4/configuration/signalling.html#debezium-signaling-enabling-kafka-signaling-channel

But when I registered the source connector configuration and sent the Kafka Signal, nothing happened, no datas were snapshotted, none of a single log appeared except for

connect                  | 2024-05-13 06:43:40,834 INFO   ||  Requested 'INCREMENTAL' snapshot of data collections '[testDB.dbo.orders]' with additional conditions '[AdditionalCondition{dataCollection=testDB.dbo.orders, filter='product_id>102'}]' and surrogate key 'id'   [io.debezium.pipeline.signal.actions.snapshotting.ExecuteSnapshot]

With signal channel set to "source", I was still able to do ad-hoc snapshot using signaling table. When I switch to use Kafka signaling method and send the kafka signal, the signaling table doesn't append any record about snapshotting.

I don't know which step did I miss ?

Here's my steps to perform snapshotting using kafka signal:

  1. Create signaling table
CREATE TABLE debezium_signal (id VARCHAR(42) PRIMARY KEY, type VARCHAR(32) NOT NULL, data VARCHAR(2048) NULL);
  1. Config source connector:
  "signal.data.collection": "testDB.dbo.debezium_signal",
  "signal.enabled.channels": "source,kafka",
  "signal.kafka.topic": "kafka-dbz",
  "signal.kafka.bootstrap.servers": "****",
  1. Send the signal message:
echo 'CDC:{"id":"d139b9b7-7777-4547-917d-e1775ea61d46","type":"execute-snapshot","data": {"data-collections": ["testDB.dbo.orders"], "surrogate-key": "id", "type": "INCREMENTAL", "additional-conditions": [{"data-collection": "testDB.dbo.orders", "filter": "product_id>102 and product_id<107"}]}}' | docker-compose exec -T kafka /kafka/bin/kafka-console-producer.sh --bootstrap-server kafka:9092 --topic kafka-dbz --property "parse.key=true" --property "key.separator=:"
Solution

  • This is an official answer, I did check the signal collection for CDC enable mode, turns out I forgot to enable CDC for it. Therefore, the signal message was not able to trigger any ad-hoc snapshot. Just check for the table CDC enable mode and it'll work just fine.