I have two python scripts, one is being executed with mpi within another script using os module. Contents of both the scripts are shown below. Let package x be the one which is causing trouble and package abc be a random package. I have added commands "type mpirun" and "mpirun hostname" to show what is changing.
runscript.py:
import os
# importing a normal package abc
import abc
os.system("type mpirun")
os.system("mpirun hostname")
os.system("mpirun -n 5 python hello.py")
# importing package x
import x
os.system("type mpirun")
os.system("mpirun hostname")
os.system("mpirun -n 5 python hello.py")
hello.py:
from mpi4py import MPI
import x
comm = MPI.COMM_WORLD
print(comm.rank)
Output of the command "python runscript.py":
mpirun is /home/pavan/packages/openmpi-4.0.7/opt-gfortran/bin/mpirun
skynet
skynet
skynet
skynet
skynet
skynet
skynet
skynet
skynet
skynet
2
3
4
1
0
mpirun is /home/pavan/packages/openmpi-4.0.7/opt-gfortran/bin/mpirun
As can be seen in output, if I import a specific package, the "mpirun hostname" doesn't print anything and hello.py script is also not executed but "type mpirun" is printing the same location for openmpi. I don't know why importing that specific package causes this problem. This doesn't happen when I import a random package abc.
Also, note that script that is being executed by mpirun is also importing that specific package x, but the script is executed properly. The issue only happens in the runscript.py
Any help on this strange error will be appreciated!! Also, what does it mean when "mpirun hostname" command doesn't output anything?
I suspect the root cause is package x imports MPI (from mpi4py).
Under the hood, that means that your program first start as a sequential program, and it is hence valid to system("mpirun ...").
Once MPI is imported (indirectly via package x), runscript.py becomes a MPI singleton and is no more allowed to `system("mpirun ...").
Before import x, you can insert the following lines
import mpi4py
mpi4py.rc.initialize=False
mpi4py.rc.finalize=False
and see if it helps.
Note that this works for me with a simple test, but that will "broke" package x if it (indirectly) issues MPI calls such as accessing MPI.COMM_WORLD.rank.
The program below illustrates this.
import os
os.system("mpirun -n 1 hostname")
print('after mpi4py')
# make sure `MPI_Init()` is not invoked under the hood
import mpi4py
mpi4py.rc.initialize=False
mpi4py.rc.finalize=False
from mpi4py import MPI
# since `mpi4py` was not initialized, the following call will crash the program if uncommented
#print(MPI.COMM_WORLD.rank)
os.system("mpirun -n 1 hostname")