Background:
I have a multi-processed NodeJS app, that does some heavy CPU processing. The app uses the child_process
module, to fork additional subprocesses.
The logic is:
main loop -> fork a Master Worker -> The MasterWorker forks up to N workers -> N workers do processing -> MasterWorker collects results -> posts back to main loop.
N here is always large enough, and greater than the number of CPU cores. Let's say that N=100, # of cores=20, and that a single worker uses 20-25% of a core. This results in 100% CPU usage for a very long time.
Obviously, I can just set N=# of Cores-1, but this limits my throughput.
Setting it to (5*(# of Cores)) -1 is still not perfect, as it depends on what other stuff the machine is doing.
My question:
I would like to keep 1 core totally "free", in order to not bring the machine to its knees while this app runs.
Is there another way that doesn't involve changing the number of worker processes?
I am aware of this: https://stackoverflow.com/a/12522859/1543677, but it's not really doable for the subprocesses the app spawns.
Really simple after all (thanks @AKX).
var nc = require('nodeaffinity');
nc.setAffinity(262143); //Cores #0-#17, modify to your liking