[SOLVED] MPI_Comm_spawn fails with "All nodes which are allocated for this job are already filled"

MPI_Comm_spawn fails with "All nodes which are allocated for this job are already filled"

I'm trying to use Torque's (5.1.1) qsub command to launch multiple OpenMPI processes, one process per node, and having each process launch a single process on its own local node using MPI_Comm_spawn(). MPI_Comm_spawn() is reporting:

All nodes which are allocated for this job are already filled.

My OpenMPI version is 4.0.1.

I am following the instructions here to control the mapping of nodes.

Controlling node mapping of MPI_COMM_SPAWN

using the --map-by ppr:1:node option to mpiexec, and a hostfile (programatically derived from the ${PBS_NODEFILE} file that Torque produces). My derived file MyHostFile looks like this:

n001.cluster.com slots=2 max_slots=2
n002.cluster.com slots=2 max_slots=2

while the original ${PBS_NODEFILE} only has the node names, and no slot specifications.

My qsub command is

qsub -V -j oe -e ./tempdir -o ./tempdir  -N MyJob   MyJob.bash

The mpiexec command from MyJob.bash is

mpiexec --display-map --np 2 --hostfile MyNodefile --map-by ppr:1:node <executable>.

MPI_Comm_spawn() causes this error to be printed:

Data for JOB [22220,1] offset 0 Total slots allocated 1   <=====

 ========================   JOB MAP   ========================

 Data for node: n001    Num slots: 1    Max slots: 0    Num procs: 1
        Process OMPI jobid: [22220,1] App: 0 Process rank: 0 Bound: socket 0[core 0[hwt 0]]:[B/././././././././.][./././././././././.]

 =============================================================

All nodes which are allocated for this job are already filled.

There are two things that occur to me: (1) "Total slots allocated" is 1 above, but I need at least two slots available. (2) It may not be right to try to specify a hostfile to mpiexec when using Torque (though it is derived from the Torque hostfile ${PBS_NODEFILE}). Maybe my derived hostfile is being ignored.

Is there a way to make this work? I've tried recompiling OpenMPI without Torque support, hopefully preventing OpenMPI from interacting with it, but it didn't change the error message.

Solution

Answering my own question: adding the argument -l nodes=1:ppn=2 to the qsub command reserves 2 processors on the node, even though mpiexec is launching only one process. MPI_Comm_spawn() can then spawn the new process on the second reserved slot.

I also had to compile OpenMPI without Torque support, since including it causes my hostfile argument to be ignored and the Torque-generated hostfile to be used.