Thread Sanitizer and Intel OpenMP

We are using GCC and Intel OpenMP.

I know that the combination of GCC and GCC OpenMP needs a special build of OpenMP (to use pthread interfaces rather than directly use the futex system call).

Is it possible to use Thread Sanitizer and GCC + Intel OpenMP? Obviously I can't just simply recompile the Intel OpenMP library.

I've started a test and I'm getting errors like

WARNING: ThreadSanitizer: data race (pid=15551)
  Atomic read of size 1 at 0x7fffec302980 by thread T4:
    #0 pthread_mutex_lock ../../../../libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4250 (libtsan.so.0+0x52d1a)
    #1 __kmp_resume_64 <null> (libiomp5.so+0xb7ae3)

  Previous write of size 1 at 0x7fffec302980 by thread T7:
    #0 pthread_mutex_init ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:1227 (libtsan.so.0+0x4c343)
    #1 __kmp_suspend_64 <null> (libiomp5.so+0xb587b)

  Thread T4 (tid=15590, running) created by main thread at:
    #0 pthread_create ../../../../libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x5ec85)
    #1 __kmp_create_worker <null> (libiomp5.so+0xb1a34)

I kind of assume that anything with the __kmp prefix should be safe, and this error is because Thread Sanitizer does not encapsulate any __kmp functions.

Solution

First, TSan by it's own does not understand OpenMP synchronization like barriers or task dependencies. We developed libarcher as part of LLVM to feed this information into TSan. If you compile a code with clang -fopenmp -fsanitize=thread, libarcher will automatically be loaded and support TSan. (verify successful loading by exporting export ARCHER_OPTIONS=verbose=true). libarcher needs OMPT support in the OpenMP runtime, which libgomp does not provide yet.

libarcher also works with newer Intel OpenMP runtimes (2020+), by explicitly loading it as an OMPT tool: export OMP_TOOL_LIBRARIES=<path-to>/libarcher.so To get libarcher you can build from source. The source code is contained in the LLVM/OpenMP project: https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.6/openmp-16.0.6.src.tar.xz

tar xf openmp-16.0.6.src.tar.xz
cd openmp-16.0.6
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DOPENMP_ENABLE_LIBOMPTARGET=False -DCMAKE_INSTALL_PREFIX=$PWD/install
make -j install

should install the code to openmp-16.0.6/build/install/, so libarcher should be in openmp-16.0.6/build/install/lib/libarcher.so

Second, for the issue of false reports from OpenMP runtime code, I suggest

export TSAN_OPTIONS='ignore_noninstrumented_modules=1'

This will disable the detection of any data race caused from non-instrumented code. So, if you link to other non-instrumented libraries (e.g. BLAS) they will also be ignored. The picture TSan gets for such libraries is limited to intercepted libc functions anyways.