mongodbapache-kafkaspring-cloud-stream

How to customize Spring Cloud Stream Schema Registry to work with No SQL backend


I am trying to integrate Spring Cloud Stream Schema Registry with MongoDB as the Schema Registry backend. I just want to know with the current implementation will this be feasible and what kinds of changes will be required to achieve this?

I have gone through the Spring Cloud Stream Schema Registry Core part and it seems lot of core changes needs to be done to achieve this.


Solution

  • We wanted to utilize one Database for the overall architecture Mongo is the one we have chosen already

    MongoDB is not a one-size-fits-all database, regardless of how they market their product.

    Besides, Confluent Schema Registry, or the native H2 database of the Spring Registry don't require any extra infrastructure, if that's your main concern. You could run the Confluent Registry directly on the brokers, in fact.

    (I don't think the H2 database is persistent, or will scale across nodes for high availability, so I wouldn't personally pick it).

    If you're concerned about Confluent Licensing, there's other options like Apicurio, or Aiven Karapace, or AWS Glue Registry...

    with the current implementation will this be feasible

    The documentation is pretty explicit about your options of:

    1. H2
    2. MySQL (or compatible)
    3. PostgreSQL (or compatible)
    4. Confluent Schema Registry (or compatible)

    MongoDB is not "compatible" with any clients for those databases.

    what kinds of changes will be required to achieve this

    You need a Collection schema for Mongo that can support, at the very least

    1. Get-by-id
    2. Get by "subject name"
    3. Get by "version" of a "subject", where version is a non negative int or "latest"
    4. Create "version" for "subject" that returns an ID

    This can be built into a Kafka client library itself, it doesn't have to be "plugged into" any Registry product; that's primarily what all the Registries are providing - a native Kafka Serializer and Deserializer implementation around a simple REST or database client. (all mentioned options are open source, go take a look at implementation)