architecturerabbitmqmicroservicesdevopseventual-consistency

Handling realtime data sync between microservices with rabbitmq queues


We have setup our microservice to use rabbitmq queues to process any upcoming data.

So, we have service A, service B, service C and service D.

When request comes to create a user to service A, it creates a user and pushed an event to newUserIsCreated queue.

Now, service B and service C needs to update their database for the newly added user.


Problem:

What if user creation fails in any service B and service C.

Possible failure scenarios can be validation failure, or service B and service C is down.

There is also issue if service D tries to access the same user.

What would service B and service C response with? Because by then they won't have the newly created user.


To sum up,

How to handle validation failure or any kind of failure between these services?

What to do if real-time syncing is required? (queues might take time to be processed)


Solution

  • You could have B and C (and any other service which needs to acknowledge that the user is created) publish an acknowledgement message to RabbitMQ to be picked up by A; A then does not acknowledge the user was created until it receives the acknowledgements. This is not foolproof: there's every chance that it succeeded everywhere but A didn't get the acknowledgements in time. For that (ideally rare) situation, you can have a process to clean up.

    It's worth noticing though, that this sort of synchronous relation is basically reimplementing request/response via a message queue, so if going down this path, why not just do direct request/response (e.g. HTTP or gRPC)?

    If you have an operation which needs to affect multiple services, it's often useful to model it as a saga, which will make requests to the various services and handle failures (and potentially give up if there have been too many failures, rolling back the changes it's made). By persisting the state of the saga (e.g. updated B, but C has not yet acknowledged the update), it can be queried.

    If you find that most operations in B are part of sagas touching A, then it may be worth considering whether B and A should be the same service, which would enable stronger consistency guarantees and also likely simplify things.