pythoncmultiprocessingctypesfile-not-found

Why does adding multiprocessing prevent python from finding my compiled c program?

I am currently looking to speed up my code using the power of multiprocessing. However I am encountering some issues when it comes to calling the compiled code from python, as it seems that the compiled file disappears from the code's view when it includes any form of multiprocessing.

For instance, with the following test code:

#include <omp.h>

int main() {
    int thread_id;
#pragma omp parallel
    {
        thread_id = omp_get_thread_num();
    }
    return 0;
}

Here, I compile the program, then turn it into a .so file using the command

gcc -fopenmp -o theories/test.so -shared -fPIC -O2 test.c

I then attempt to run the said code from test.py:

from ctypes import CDLL
import os

absolute_path = os.path.dirname(os.path.abspath(__file__))
# imports the c libraries
test_lib_path = absolute_path + '/theories/test.so'
test = CDLL(test_lib_path)
test.main()
print('complete')

I get the following error:

FileNotFoundError: Could not find module 'C:\[my path]\theories\test.so' (or one of its dependencies). Try using the full path with constructor syntax.

However, when I comment out the multiprocessing element to get the follwing code:

#include <omp.h>

int main() {
    int thread_id;
    /*
#pragma omp parallel
    {
        thread_id = omp_get_thread_num();
    }
    */
    return 0;
}

I then have a perfect execution with the python program printing out "complete" at the end.

I'm wondering how this has come to happen, and how the code can seemingly be compiled fine but then throw problems only once it's called from python (also I have checked and the file is in fact created).

UPDATES:

  1. I have now checked that I have libgomp-1.dll installed
  2. I have uninstalled and reinstalled MinGW, with no change happening.
  3. I have installed a different, 64 bit version of gcc and, using a different (64 bit python 3.10) version of python have reproduced the same error. This also has libgomp-1.dll.
Solution

  • I think this has the same root cause as [SO]: Can't import dll module in Python (@CristiFati's answer) (also check [SO]: PyWin32 and Python 3.8.0 (@CristiFati's answer)).

    A .dll (.so) is only loaded when its dependencies are successfully loaded (recursively).
    [SO]: Python Ctypes - loading dll throws OSError: [WinError 193] %1 is not a valid Win32 application (@CristiFati's answer) focuses on a different error, but covers this topic.

    When commenting out omp_get_thread_num call, the linker no longer links test.so to libgomp*.dll (as it doesn't need anything from it), and code runs fine (all required .dlls are found).

    To fix the issue, you should add to os.add_dll_directory (before attempting to load the .so):

    1. libgomp*.dll's directory

    2. MinGW's bin directory

    3. Any other directory that may contain .dlls that are required (dependents)

    To see a .dll dependencies, check [SO]: Discover missing module using command-line ("DLL load failed" error) (@CristiFati's answer).

    Notes: