rabbitmqpikarabbitmq-exchangerabbitmq-shovel

RabbitMQ Messages not getting sent to dead letter after first time

We've got our queues configured to send dead letter messages (nack'ed messages specifically) to a dead letter exchange that routes them by their original topic to individual dead letter queues. This all works great and when messages are nack'ed they're sent to the correct dead letter queue.

The trouble comes in when we shovel those messages back from the dlq to the normal queue, where they get nack'ed again. For some reason, this second time through they just disappear instead of being sent back to the dead letter exchange.

I assume there's some sort of "circular message routing" detection going on, but can't find anything like that. Inspecting the messages the second time through gives all the expected headers so I'm not sure what such a thing could even be based on. Any suggestions of where to look next or if rabbit has such a thing would be greatly appreciated!

If it's necessary, our consumers are written in python using the pika library for communication.

Solution

  • Assuming that you have the following queues/exchanges:

    Exchanges

    Queues

    Bindings

    Finally, I assume that you are using the shovel plugin on the queue.dlq management page like this, to move messages from queue.dlq into queue:

    enter image description here

    Here is how the routing works when you send a message with test_message as the routing_key to the global_exchange:

    1. Message lands in queue from the binding on test_message
    2. Consumer nack's (nack or reject doesn't matter) the message, thus dead-lettering it
    3. The x-dead-letter-exchange argument sends it to DLX with routing_key= test_message
    4. Because of the queue.dlq binding, that queue receives the message

    When you use that particular management panel to shovel messages back into queue, it uses the default exchange. This changes the routing key. So the 2nd receipt of the message has a routing key that is equal to the name of the queue you are shoveling into.

    Since you do not have an x-dead-letter-routing-key configured, the message is dead-lettered to the current routing key:

    If this is not set, the message's own routing keys will be used.

    So on the result of the shovel, this is how it is routed:

    1. Message appears in queue with routing_key = queue
    2. Since there is no x-dead-letter-routing-key configured, it dead letters to DLX with routing_key = queue
    3. No binding to queue in DLX, message dropped

    There 2 potential workarounds:

    1. Add another binding to queue.dlq to routing_key = queue
    2. Manually configure the x-dead-letter-routing-key on queue to always send to the same routing key on dead-letter no matter what message was originally sent to it and ensure there is a binding to it within DLX