ipython-parallel

which core runs the session during concurrent computing


I opened a ipython session, started ipcluster with 4 engines, let engine[0] continuously work in a non-blocking mode.

Now I have some computation to do in the ipython session. How do I know and make sure that the computation in this session is not using the same core/cpu/resources as engine[0]?


Solution

  • Short answer: If you don't send the work to engine 0 explicitly, it won't use engine 0's resources.

    IPython doesn't manage cores or other physical resources, it just allocates processes. It is left to your operating system to assign concurrent processes to your CPU cores. If you have started an IPython cluster with four engines and are using it from an interactive IPython session, you have five Python processes that can run code concurrently—your four engines, plus the interactive session itself. If engine 0 is running a job in the background and you perform local computations in your interactive session, your operating system should assign the work done in the interactive session to a different core than the one occupied by engine 0, assuming there is a core available.