pythonmpi4pyipython-parallel

ipyparallel with mpi cannot find engines


I'm trying to get a basic ipyparallel environment working using mpi4py as described in the ipyparallel documentation. After starting the ipcluster, I load ipython and try to create a client but it has no IDs and accessing the directview returns a NoEnginesRegistered exception.

Steps I take to get to this point:

  1. Create a new environment: conda create --name=ipyparallel and source activate ipyparallel
  2. Install ipyparallel and mpi4py: conda install ipyparallel mpi4py
  3. Create a new ipython profile: ipython profile create --parallel --profile=mpi
  4. Edit ~/.ipython/profile_mpi/ipcluster_config.py and add c.IPClusterEngines.engine_launcher_class = 'MPIEngineSetLauncher'
  5. Launch cluster with ipcluster start --profile=mpi

Then I launch ipython and run the following:

import ipyparallel as ipp
c = ipp.client(profile="mpi")
c[:] # <-- NoEnginesRegistered exception

Step 5 reports that "Engines appear to have started successfully" and I can see that a process named "mpiexec" is running. Strangely, I tried these same steps on another machine with the same OS and there it worked with no problems. What am I missing?


Solution

  • I solved the problem for anyone coming here with a similar problem. During installation I had added the notebook extension to jupyter's global config. Not sure why that caused this problem but it's fixed now. Outside of a conda environment, I ran:

    sudo pip install ipyparallel
    sudo jupyter nbextension disable --py ipyparallel
    sudo jupyter nbextension uninstall --py ipyparallel
    sudo pip uninstall ipyparallel
    

    and then inside the conda environment I can connect to the ipyparallel engines.