node.jsredisioredis

BLPOP blocks my redis connection when using it in multiple running queues


I have 8 queues running and I want to pop if the queue has a push. The application has multiple users and the queues are almost always having data.

I want to replace LPOP with BLPOP but using it in all the queues doesn't pop the data

here is a sample of my code

async function queuePop(queue) {
    const value = await redis.blpop(queue, 0);
    if (value) {
        const data = JSON.parse(value[1]);
        return data;
    }
}

async function processQueue(queue) {
    while (true) {
        try {
            const data = await popFromQueue(queue);
        } catch (err) {
            console.error('Error processing job', err);
        }
    }
}

function ProcessingFunction(data) {
    const data = await processQueue('queueName')
    console.log(`Processing data: ${data}`);
}

calling ProcessingFunction() in a setTimeout

Solution

  • To implement BLPOP for multiple queues, we need multiple instances of Redis Client. For example:

    const queues = { name: 'queue1', client: null, ... }
    // assigning the instance of the Redis client to all the queues in the queues
    

    and Thus when there is no data in the queue1, the connection will drop for that instance and the same goes for all instances of Redis client.

    There is a limitation for BLPOP as one cannot use Limit like we do in LPOP