How to apply Debezium CDC events from Pub/Sub onto a database?
Currently, I am setting up an environment to test CDC (Change Data Capture) with Apache Debezium for replicating database tables from two databases to one. The CDC events are captured and sent to Google Pub/Sub topics, but I have no idea of how to propagate those messages to the target database. I am ultimately trying to find the missing piece in the puzzle that is highlighted as a question mark in the image down below. How can I connect my Google Pub/Sub topic messages to my target database (DB C)?
DB A and DB B are both PostgreSQL database instances and the database system for DB C is not determined.
Each message in my Pub/Sub topics contains a Debezium CDC event.
Also, I am planning to stick with Google Pub/Sub as a message broker if possible.
The idea of debezium is to take data at rest (DB A and DB B) and get it into motion. It uses Change Data Capture for it. Once a target messaging system Kafka / Kinesis / Event Hub / Pub Sub is chosen the data is then sent to a set of topics on this messaging system
From this messaging system , there has to be some application that will read this data and drop this data into a Sink (DB-C). This could be any application that is capable of reading from the messaging applications and having connectivity to DB-C . Modern realtime systems are capable of that. Cloud providers also give options for it. Examples of these include systems such as Storm , Spark , Flink etc that can consume off these topics and then persist data. Since it is Google pub-sub , there may be some options to read off it natively as well
- Ad-Hoc Snapshot with Debezium SQL Server Source Connector using Kafka Signal doesn't work
- How do I run debezium with Confluent platform?
- Debezium docker with connect table.include.list not working
- Debezium fails to reconenct after lost connection to DB
- Debezium KafkaConnect, kafka_connect_run.sh No such file or directory
- Flink failed to deserialize JSON produced by Debezium
- Postgres Debezium does not publish the previous state of a record
- How Kafka Connect Cluster is operated in production (number, distribution, etc.)
- Pyspark stream kafka debezium topic Error format, ETL
- Can 2 Debezium Connectors read from same source at the same time?
- How to connect to a MongoDB over SSH from Debezium
- debezium standalone server with http sink
- Kafka Connect Transformations - RegexRouter replacement topic names in lowercase
- Why Debezium Connector can't connect to a SASL activated broker?
- Kafka connector giving error when trying to connect oracle DB debezium
- Debezium Server with NATS JetStream No Responders error
- Connection Timeout: Cant connect mongo with debezium for CDC
- DebeziumException "The connector previously stopped while taking a snapshot, but now the connector is configured to never allow snapshots"
- Error creating CDC Azure Database MySQL to Apache Kafka
- Debezium setup with Rabbit MQ and MS SQL SERVER
- use debezium to capture the existing data (full refresh)
- How to tell debezuim Mysql source connector to stop retaking snapshots of existing tables in kafka topic?
- How does Debezium read the binlog?
- Debezium signal table at connector startup
- Debezium with Postgresql RDS stuck at Creating replication slot with command CREATE_REPLICATION_SLOT
- How to perform Ad-Hoc Snapshot with Debezium SQL Server Source Connector
- Debezium does not capture 'cascade delete'
- Debezium-JDBC connector handling operations using other columns aside from pk (id)
- Debezium Source Connector to Oracle Databse transform `RAW(16)` to Guid string
- Debezium connect: java.lang.NoClassDefFoundError: io/confluent/connect/schema/AbstractDataConfig