apache-kafkaapache-kafka-connectspring-cloud-streamspring-cloud-dataflowmongodb-kafka-connector

Register Kafka connect MongoDB connectors as apps in SCDF

I'll get straight to the point :

Is it possible to register the MongoDB Kafka sink and source connectors as applications in Spring Cloud Data Flow ? or other types of Kafka connectors in the matter of fact ?

MongoDB Kafka Source connector requires MongoDB to be configured as a replicaset cluster in order to read the change streams in the opLog (a standalone MongoDB instance cannot produce a change stream). Does the SCDF MongoDB source starter use MongoDb cluster's change streams to detect change events ? or does it read changes directly from the MongoDB database ?

Thanks

Solution

  • We have looked at integrating Spring Cloud Stream with Kafka connectors. There is no easy way to do it without custom code. We have a change data capture source that works with MongoDB as you describe: https://github.com/spring-cloud/stream-applications/blob/main/functions/supplier/cdc-debezium-supplier/README.adoc