architecturemessagingmq

Architecture: using a separate queue for error handling?


We have a small microservice which sole purpose is to receive messages through a queue and send these to an external system. The messages can come from any number of other services and the service has no knowledge about the content of the message. The external system can either accept or reject this message. I see a number of options to deal with the responses of the external system:

I'm inclined to choose the third option to separate concerns and not disturb a happy flow with an error flow, but would like feedback why this could be a bad choice.

Are there any resources available documenting best practice solutions for these kinds of problems?

(I am aware above could be rewritten to a service reading from the queue and storing the message in a db, asynchronously sending the messages to the external system and storing the response in a db, publishing an event to indicate the message was processed and developing an api to allow to retrieve the response from the db, but that would require an extra db, more work and more resources, which in my opinion would be overkill)


Solution

  • I would go for the 3rd option as well and separate your concerns for a couple of reasons.

    1. If all the different types of messages are on the same queue then your consumers will have to use a filter/selector to get the messages that they want. This will increase the complexity of the consumer. Also, consumer-side filters/selectors are generally discouraged as they negatively impact performance due to the queue scanning the broker has to do to find messages which match the consumer's filter.
    2. If all the messages are on the same queue then management will be more complex. For example, if you wanted to know how many errors had occurred you'd need your management tool to scan the queue for messages matching the error pattern. If the errors were on a separate queue you'd simply need to the look at the number of messages in the queue.
    3. Multiple queues are usually better for performance as they increase parallelism and reduce bottlenecks. Most modern brokers (e.g. ActiveMQ Artemis) scale very well with multiple queues & clients.

    It's worth noting that message brokers aren't meant for long-term data storage like a database is. If you intend to keep the response details around for awhile you probably want to offload them to a database at some point.