multithreadingopenmpreal-timehpcaffinity

Running OpenMP threads on isolated CPUs


How can the CPU affinity for OpenMP threads be set on a (Linux) system with cores removed from the scheduler (e.g. using the isolcpus boot parameter)?

On a system where only the first 12 cores are available to the scheduler (isolcpus=12-127), calling the OpenMP program with

GOMP_CPU_AFFINITY=64-127 OMP_NUM_THREADS=64 taskset --cpu-list 64-127 command

all 64 OpenMP threads run on CPU 64.

With

GOMP_CPU_AFFINITY=64-127 OMP_NUM_THREADS=64 command

All 64 OpenMP threads run on CPUs 0-12.

How can I get the 64 OpenMP threads to run on CPUs 64-127?

I could do it programmatically within the software using

#pragma omp parallel for
for(...)
  pthread_setaffinity_np(...)

Is there any way this can be done externally when starting the OpenMP program?

(Note: A similar issue was brought up in Core isolation limits OpenMP to a single core but no solution was provided. The given answer "I think you are asking OpenMP to do something that is impossible." is not true, as all that the OpenMP library needs to do is to call pthread_setaffinity_np() or equivalent functions. It shouldn't even need taskset to run on a CPU specified in GOMP_CPU_AFFINITY.)


Solution

  • It appears that both Intel libiomp and GNU libgomp must be intersecting the affinity mask inherited from the calling process (non-isolated cores 0-11) with any GOMP_CPU_AFFINITY, KMP_AFFINITY or OMP_PLACES specifiers. Hence taskset is needed.

    Here is what works for Intel:

    KMP_AFFINITY=explicit,proclist=[12-191] taskset --cpu-list 12-191 command
    OMP_PLACES="12:180" taskset --cpu-list 12-191 command
    OMP_PLACES="{12}:180" taskset --cpu-list 12-191 command
    

    This works for GNU:

    GOMP_CPU_AFFINITY=12-191 taskset --cpu-list 12-191 taskset --cpu-list 12-191 command # note the double taskset
    OMP_PLACES="{12},{13},(explicitly list all cores),{191}" taskset --cpu-list 12-191 command
    OMP_PLACES="{12}:180" taskset --cpu-list 12-191 command
    

    This does not work for Intel:

    taskset --cpu-list 12-191 command
    KMP_AFFINITY=explicit,proclist=[12-191] command
    OMP_PLACES="12:180" command
    OMP_PROC_BIND=true OMP_PLACES="12:180" command
    

    This does not work for GNU:

    taskset --cpu-list 12-191 command
    OMP_PLACES="12:180" taskset --cpu-list 12-191 command
    OMP_PLACES="12:180" taskset --cpu-list 12-191 taskset --cpu-list 12-191 command
    GOMP_CPU_AFFINITY=12-191 taskset --cpu-list 12-191 command
    OMP_PLACES="{12},{13},(explicitly list all cores),{191}" command
    

    OMP_NUM_THREADS=180 and isolcpus=12-191 in all examples.