I'm trying to run CULA's multiGPU example but I get the following error. Did someone try the example and manage to get it to work? I have two GPUs on board.
[xxxx@xxxxxxxxxx multiGPU]$ ./multiGPU Found 2 devices, will launch 2 threads
Thread 0 - Launched Thread 0 - Binding to device 0 Thread 1 - Launched Thread 1 - Binding to device 1 Thread 0 - CUDA Error: exclusive-thread device already in use by a different thread (see Programmer's Guide) Thread 1 - Allocating matrices Thread 1 - Initializing CULA Thread 1 - Calling culaSgeqrf Thread 1 - Shutting down CULA
Thread 0 - Failed
I was able to reproduce this problem by setting the compute mode on a 2-GPU system to EXCLUSIVE_PROCESS and running the multiGPU example from CULA.
The problem is that the multiGPU process becomes bound to the GPU and prevents one of the threads spawned by pthreads from accessing that GPU, resulting in the error messages indicated (CUDA Error: exclusive-thread device already in use by a different thread).
One solution to this problem is to set the GPUs to the "Default" compute mode so that they can be accessed by multiple threads at once.
# nvidia-smi -c 0
The program can also be run with the GPU devices in EXCLUSIVE_PROCESS compute mode, but the CUDA Multi-Process Service (MPS) must be used. This daemon will manage processes on the device so that spawned processes interact with the device as if it were in DEFAULT compute mode. For me, the program ran properly once the MPS daemon was running for the conflicted device. The directions for starting the daemon can be found in the Appendix of the MPS documentation.
https://docs.nvidia.com/deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf
-Dan