node.jsrabbitmqnode-amqplib

Getting too many channels error for a period of time

I have 16 queues & mulitple consumer servers for those queues. I have created one dedicated channel for each queue to consume messages. Consumer & dispatch channels on each server share same connection.

When I dispatch messages to each queue, I do the following:

  1. create a new channel
  2. bind channel to the queue with proper routing
  3. dispatch the message
  4. close the channel

I have lots of incoming webhooks from Shopify & these webooks contents are dispatched to specific queues.

While processing each message, I need make an API call to Shopify. Shopify API has rate limit. If I hit rate limit once, I redispatch all messages from the consumer back to rabbitmq with a delay header of 1 minute(time required to clear the API rate limit).

Now, when I have several consumers running with lots of messages in the queue & I re-dispatch those messages, I get too many channels error for a period of time. How can I avoid this error?

I tried to keep 2 dedicated channels per queue:

  1. for conusmer purpose only
  2. for dispatch purpose only

For, 16 queues, & around 11 consumer servers. This way, I always have to keep 352 channel open. This caues CPU utilization on rabbitmq host server to reach >90% which is also an issue. As the server can crash any time.

Solution

  • I found the solution to the problem after digging through the RabbitMQ documentation.

    Instead of creating a new channel for each dispatch, I created a single channel & kept it alive for the entire connection session. When creating the channel, I asserted all the exchanges that would be used by my queues.

    Then I just publish the messages to the desired exchange with the routing key. As my queues are already bonded with the exchanges & listen for messages with a given routing key, the messages end up in the correct queue!

    This way I can maintain just 01 connection & only 01 channel per server!