perlrabbitmqmessage-queueanyevent

Anyevent::RabbitMQ Perl QoS prefetch_count not working

I've been trying to use RabbitMQ perl library Net::RabbitFoot which uses AnyEvent::RabbitMQ underneath. According to RabbitMQ Tutorial, setting prefetch_count to 1 should ensure fair dispatch, as in should not dispatch a message to a worker that is already busy on another message. However, the perl implementation Net::RabbitFoot, does not seem to work that way even after setting the qos as described here, line 54. It seems to just do vanilla round-robin dispatch and ends up dispatching to machine that is already executing a job. This is the qos implementation. Could you help me with figuring out why this is happening? Is it a bug in the library?

Thanks in advance.

Edit:

This is my setup: 2 consumers attached to the same-named queue. When I dispatch a lot of messages, I see this pattern: Consumer 1: Msg1, Msg3, Msg5 ... Consumer 2: Msg2, Msg4, ... All messages are from the same queue. What happens now is if Msg3 hogs Consumer 1, still Msg5 is sent to Consumer 1 while Consumer 2 is sitting free.

Solution

  • vanilla round-robin? uh?

    The prefetch_count=1 comes useful when there are many consumers attached to the same common queue. In fact by default the client libraries will prefetch many messages in one shot.

    So the default odd effect, that you want to avoid by setting it to one, is that one client get most (or all) the messages, and other consumers get few or none, being the load unbalanced.

    However you speak of "vanilla round-robin": that happens when you have different (probably unnamed/temporary) queues attached to a direct exchange, one per consumer. But in this way you have no way to balance the load dynamically.

    If I'm guessing right you need to change your configuration and let all the consumers attach to the same named queue.

    EDIT: from the comment of the OP, this is not the case.

    Alternatively it's possible that your consumers are configured with auto-ack, or they do send the ACK before completing their job. In this case too the RabbitMQ client API thinks that it's free to get another message: you need to send the ack back only after the local task regarding that message has been completed.