pythonmultiprocessingpoolemcee

Multiprocessing the Python module 'emcee', but not all available cores on the machine are being used

I am parallelizing emcee using multiprocessing module as stated in the emcee document. However, htop shows that the program keeps using a limited number of cores (26-27). The computer I am running my code on has 80 available cores and I would like to make use of all of them to speed up the code.

Can someone please help me with this? I did not find any solution in the document.

Solution

  • In case someone runs into the same issue, I just found out that emcee will use a number of cores equal to half the number of walkers. In my case I had 50 walkers and hence, 25 CPU cores were being used.