I am running an executable (I don't have access to source code) that calls mpirun. I'm getting the following error which is common if more cores are requested than available on the CPU:
There are not enough slots available in the system to satisfy the 12
slots that were requested by the application:
/Users/me/Library/app/executable
Either request fewer slots for your application, or make more slots
available for use.
A "slot" is the Open MPI term for an allocatable unit where we can
launch a process. The number of slots available are defined by the
environment in which Open MPI processes are run:
1. Hostfile, via "slots=N" clauses (N defaults to number of
processor cores if not provided)
2. The --host command line parameter, via a ":N" suffix on the
hostname (N defaults to 1 if not provided)
3. Resource manager (e.g., SLURM, PBS/Torque, LSF, etc.)
4. If none of a hostfile, the --host command line parameter, or an
RM is present, Open MPI defaults to the number of processor cores
In all the above cases, if you want Open MPI to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.
Alternatively, you can use the --oversubscribe option to ignore the
number of available slots when deciding the number of processes to
launch.
My problem is that I cannot change the command-line options for mpirun e.g. using --oversubscribe. Rather, I need to change the default number of cores that OpenMPI "sees". (This would be an easy fix otherwise as in this case)
Is there an environment variable or something I can update to trick openMPI into working?
Ah. I found the default OpenMPI hostfile at /usr/local/etc/openmpi-default-hostfile (on Mac) and added at the end (in a new line):
localhost slots=12
So OpenMPI was reading a default slots count of 6, since I have 6 cores on my system (The error only occurred for >6 requested CPUs). But I have 12 threads and wanted to use the CPU at full capacity.
This worked for me since I was not running mpirun in the command line.