How to compile the PMI support for running Chapel/GASNet on Omni-Path networks?

I'm trying to run Chapel/GASNet on a cluster equipped with Omni-path network.

GASNet official documentation for Omni-Path recommends to use the ofi-conduit by passing --enable-ofi --disable-psm --disable-ibv. However, as I do not know where to pass this configuration, I decided to use the PSM conduit for Omni-Path.

1) I can run Chapel/GASNet using GASNET_PSM_SPAWNER='ssh'. However, this spawner is resulting in quite slow PGAS.

2) I can only use MPI as the spawner if I set -mca mtl ^psm,psm2, which is also slow. Otherwise, I receive several errors.

3) I tried to use PMI as the spawner. However, I receive the following error message: Spawner is set to PMI, but PMI support was not compiled in usage: gasnetrun...

How can I compile the PMI support and for using GASNET_PSM_SPAWNER='pmi'?

Here are my other Chapel/GASNet runtime variables:

CHPL_COMM='gasnet'

CHPL_LAUNCHER='gasnetrun_psm'

CHPL_COMM_SUBSTRATE='psm'

CHPL_GASNET_SEGMENT='everything'

CHPL_TARGET_ARCH='native'

HFI_NO_CPUAFFINITY=1

All the best,

Tiago Carneiro.

Solution

I don't have easy access to an Omni-path system to test any of this, but in the interest of trying to get you an answer:

It appears to me as though Chapel ought to build and use the ofi-conduit if you do the following:

set CHPL_COMM_SUBSTRATE=ofi in your environment (e.g., export CHPL_COMM_SUBSTRATE=ofi)
re-build Chapel (e.g., make or gmake from $CHPL_HOME)
re-compile and re-run your program

The choice of spawner/launcher that you use should not have an impact on your program's performance that I am aware of... It is simply the mechanism for getting the executables up and running on the system's compute nodes. That is, if you have a technique that is working, I'd suggest sticking with it rather than trying to use other spawners/launchers (In any case, I'm not personally familiar with how to use the PMI spawner and am fairly certain that Chapel doesn't currently have a launcher that wraps it).

By contrast, the choice of conduit can have a very large impact on program performance, as it governs how communication takes place throughout the program's execution.

As a reminder: As with any Chapel program, once you have it working correctly and are doing performance studies, be sure to use the --fast flag.