amazon-web-servicesamazon-sqsterraform-provider-awsaws-event-bridgesqslistener

How to make multiple consumers receive the same message from a single AWS SQS Queue?


I have an AWS SQS queue that is subscribed to an SNS topic. The queue currently has two consumers:

  1. A Java class using @SqsListener from the io.awspring.cloud.sqs.annotation package.
  2. An EventBridge pipe configured to use the same SQS queue as its source.

(Using terraform, Java)

The issue is that at any given time, only one of these consumers receives a message from the queue. I understand that SQS is designed to deliver each message to only one consumer. However, I want both consumers to process the same message independently.

Questions:

  1. Is there a way to configure SQS, EventBridge pipes, or @SqsListener to allow both consumers to receive the same messages independently?
  2. If not, what is the best architectural workaround to achieve this with minimal changes and only one SQS queue?

Code:

    @SqsListener("${app.consumer.shopper-order.queue}")
    public void listen(String message) throws JsonProcessingException {
      // do stuff
    }

Any help or guidance would be greatly appreciated!


Solution

  • The correct architecture would be:

    This way, each message is sent to both queues and each consumer can independently consume the messages at their own pace. It is then easy to add additional consumers in future.

    See: Fanout Amazon SNS notifications to Amazon SQS queues for asynchronous processing - Amazon Simple Notification Service

    Alternatively, send messages to an Amazon Kinesis stream and have the consumers pull messages from the stream. Multiple consumers can read from the same stream, each with their own 'current pointer'. They can also replay messages rather than deleting them after consumption.

    Consuming from an SQS queue is easier (and cheaper) than consuming from a Kinesis stream.