[SOLVED] SQS Messaging Design: Payload vs. Reference?

SQS Messaging Design: Payload vs. Reference?

I'm working on an event-driven system using AWS SQS and facing a design decision when sending commands between services. The command triggers an operation in the receiving service, and that operation requires multiple pieces of data — potentially 15+ parameters.

The dilemma:

Should the SQS message contain all the necessary information for the command to be processed (i.e. fully self-contained payload)? OR
Only include a reference ID, so that the receiving service can fetch the data it needs (e.g. from a shared database or API)?

I’m trying to keep the architecture clean, scalable, and maintainable

My gut: It feels messy to cram all parameters into a single message, but I'm aware of introducing latency and tight coupling by forcing the consumer to retrieve external data.

What’s the most robust and clean approach in your experience? Any best practices for hybrid strategies?

Solution

I'd always prefer including the necessary data in the message and avoid the "callback". Only sending a notification with a reference, and have the receiver get the actual data via query

introduces an unnecessary runtime coupling
makes the receiver more complex, it now has to handle more failure cases - e.g. if the service providing the data is unavailable, the receiver needs to stash the command for later retry.
can cause load peaks on the sender / data provider
decouples the command from the data - maybe the parameter values change between the time the command has been issued and the callback, leading to possible confusion.

I'd only use a notification with a reference if the payload is too big for the message channel and if the referenced data is immutable. E.g. if your service generates a PDF document and wants to notify others about it.

P.S.: You should be mindful of the terminology. Your saying you're developing an event- driven system, but then you talk about sending commands.

Events and commands are very different things!

Events describe something that happened in the past, a fact, and are fire-and-forget. An event can't "fail".
Commands are a request to perform an operation or change a state, and are request-reply, they yield the response if the command was executed successfully or not. Commands can fail.

(In almost all cases... for a more differentiated view see https://www.reactivesystems.eu/2024/11/30/not-all-commands-are-equal.html)

If you're sending commands, you're implementing a message-driven system, but it's not an event-driven system.
In fact, if you want to keep it scalable and maintainable, you should think about making it actually event-driven, and publish events, not send commands.