node.jscluster-computingpm2

How to perform multiple workers to do HTTP request tasks in node js


while (true) {
    const requestIdArray = await db.find()

    # I want to perform the following functions (inside for loop) parallell to increase the efficiency
    for (let i = 0; i < 10; i++ ) {
        const workingIds = requestIdArray.slice(i*10, i*10+10)
        const apiRes = await thirdPartyAPIRequest(workingIds)
        await work(apiRes) // (20% CPU intense, 40% DB CRUD, 40% HTTP request)
    }
}

I want to spread the workload instead of using for loop to perform almost same tasks work() every time, what's difference is requestIdArray can be different depends on what I get from the database.

I don't know whether I should use native cluster, or pm2 to achieve it, and how, since most of the tutorial I can find online is about the backend server load balancing.


Solution

  • while (true) {
        const requestIdArray = await db.find()
    
        # I want to perform the following functions (inside for loop) parallell to increase the efficiency
        const workLoad = Array.from({length: 10}, (x, i) => i)
             .map(async i => 
             {
                 const workingIds = requestIdArray.slice(i*10, i*10+10)
                 const apiRes = await thirdPartyAPIRequest(workingIds)
                 await work(apiRes) // (20% CPU intense, 40% DB CRUD, 40% HTTP request)
             });
        await Promise.all(workLoad);
    }