azure-cosmosdbazureservicebusoutbox-pattern

How to implement outbox pattern in Cosmos DB


I'm looking to implement support for the outbox pattern in Cosmos DB.

However, Cosmos DB doesn't seem to support transactions across collections.

Then how do I do it?

I've been considering a few approaches to implement this:

Use Service bus transactions

Within a Service bus transaction scope, send the message (not committed just yet), do the Cosmos DB update and, if it works, then we commit the service bus transaction to have the message made available to subscribers.

Use triggers to insert rows in the outbox collection

As inserts/updates happen, we use Cosmos DB triggers to insert the respective messages into the outbox table and from then on, it's business as usual.

Use triggers to execute azure functions

Create Azure functions as Cosmos DB triggers. I almost like this but it would be so much better to get a message straight to service bus.

Use a data pump

Add two fields UpdateTimestamp and OutboxMessageTimestamp. When a recorded is updated so does the UpdateTimestamp.

Some process looks for records in which these two don't match and for each of those creates a notification message and relays it to the respective queues or topics.

Of course, then it updates the second timestamp so they match.

Other ideas on how to do this?


Solution

  • in general, you store things in your cosmos db collection. then you have change feed sending these changes to some observer (lets say azure function). then your azure function can do whatever: put it in queue for other consumers, save into another collection projected differently, etc... within your azure function you should implement your dead letter queue for failures that are not related to function runtime (for example, writing to another collection failed due to id conflict)

    [UPDATE]

    Let me add a bit more as a response to your comment. From my experience, doing things atomically in distributed systems boils down to:

    1. Always do things in same order
    2. Make second step itempotent (ensuring you can repeat it any number of times getting same result)
    3. Once first step succeeded - repeat second step until successful

    So, in case you want to send email upon something saved into cosmos db, you could:

    1. Save record in cosmos db
    2. Have azure function listen to change feed
    3. Once you receive inserted document > send email (more robust solution would actually put it in queue from which some dedicated consumer sends emails)

    Alternative would be to have initial command (to save record) put in queue and then have 2 consumers (one for saving and one for sending emails) but then you have a problem of ordering (if thats important for you).