node.jsmultithreadingaffinity

Can I set the affinity of a multi-processed Nodejs application?


Background:


I have a multi-processed NodeJS app, that does some heavy CPU processing. The app uses the child_process module, to fork additional subprocesses.

The logic is:

main loop -> fork a Master Worker -> The MasterWorker forks up to N workers -> N workers do processing -> MasterWorker collects results -> posts back to main loop.

N here is always large enough, and greater than the number of CPU cores. Let's say that N=100, # of cores=20, and that a single worker uses 20-25% of a core. This results in 100% CPU usage for a very long time.

Obviously, I can just set N=# of Cores-1, but this limits my throughput.

Setting it to (5*(# of Cores)) -1 is still not perfect, as it depends on what other stuff the machine is doing.


My question:

I would like to keep 1 core totally "free", in order to not bring the machine to its knees while this app runs.

Is there another way that doesn't involve changing the number of worker processes?

I am aware of this: https://stackoverflow.com/a/12522859/1543677, but it's not really doable for the subprocesses the app spawns.


Solution

  • Really simple after all (thanks @AKX).

    1. Install nodeaffinity
    2. At the top of each worker, use:

    var nc = require('nodeaffinity');
    nc.setAffinity(262143);    //Cores #0-#17, modify to your liking