pythonpython-importctypes

Paradoxon: silent crash on Python's ctypes.CDLL when importing, but not when running directly - how is this possible?

So, being a Linux guy I stumbled into something pretty puzzling on Windows that I just can't explain.

I have a project structure analougus to this example:

D:\PROJECT
|
|   tolkien.py
|   __init__.py
|   
\---MiddleEarth
    |   gondor.py
    |   isengrad.c
    |   __init__.py
    |   
    \---lib
            isengrad.so

Problem: I compile isengrad.c into the shared libary isengrad.so, then load it in gondor.py. My aim is to import gondor.py into tolkien.py.
While gondor.py runs flawlessly when it is run directly, when I import it, the code exits at the point when I load the shared library via ctypes.CDLL, without any error messages.

Reproduction: The content of the files (added some "status messages" to follow where the problem happens):

isengrad.c: 

int isengrad(int hobbit){
    return hobbit/2;
}

This is then compiled to isengrad.so with 

D:\project>chdir MiddleEarth
D:\project\MiddleEarth>gcc -fPIC -shared -o lib/isengrad.so isengrad.c

The shared library is then accessed in gondor.py:

print("started gondor")

import os, ctypes
path_to_isengrad = "D:/project/MiddleEarth/lib/isengrad.so"  

print("gondor loads isengrad")
gondor = ctypes.CDLL(path_to_isengrad)     # <--- crashes here when imported, not when ran directly
print("gondor loaded isengrad")


gondor.isengrad.argtypes = (ctypes.c_int,)

def faramir(hobbit):
    catched_hobbits = gondor.isengrad(hobbit)
    return catched_hobbits

if __name__ == '__main__':
    print(faramir(5))
    print("gondor ran")

print("gondor finished")

which is then imported in tolkien.py:

print("started tolkien")
from MiddleEarth import gondor
print("tolkien imported gondor")

got = gondor.faramir(4)
print(got)

print("tolkien worked")

Now check what happens when I use gondor.py directly VS when I import it in tolkien.py:

D:\project>python MiddleEarth/gondor.py
started gondor
gondor loads isengrad
gondor loaded isengrad
2
gondor ran
gondor finished

D:\project>python tolkien.py
started tolkien
started gondor
gondor loads isengrad

D:\project>

Directly running it causes no problem at all. But importing it causes the whole thing to crash without any word and traceback when loading the shared library. How is this even happening? I even hard-coded the path to the shared library, so different working directory shouldn't be a problem... I didn't have any problem with the very same project on Kubuntu, so this is probably some Windows-related stuff.

Environment:

Solution

  • From the moment I saw this question, I wanted to say it's Undefined Behavior (UB). Python comes with its C runtime (UCRTLib), while the Cygwin .dll comes with its own. Mixing compilers and C runtimes in a process, is generally a recipe for disaster.
    I found an official statement [Cygwin]: 6.15. Can I link with both MSVCRT*.DLL and cygwin1.dll? (emphasis is mine):

    No, you must use one or the other, they are mutually exclusive.

    Check [SO]: How to circumvent Windows Universal CRT headers dependency on vcruntime.h (@CristiFati's answer) for more details on MSVCRT*.DLL (VCRuntime*.dll).

    Now, the beauty of UB is that it describes a seemingly random behavior.

    I've prepared a comprehensive example (slightly modifying your code).

    Outputs:

    As seen, cross compiler .exe - .dll worked in 7 (out of 8) cases (crashed on 064bit Win Python with code01.py), while the same compiler worked in all 8 of them.

    So, I'd advise that when playing with such environments, try to keep the compilers used to build various parts consistent (or compatible at least).

    Similar question:


    Update #0

    I just thought of a reason why things could go wrong on 064bit: sizeof(long) generally differs (sizes below are in bytes):

    Same thing for sizeof(long double) (which is generally 2 * sizeof(long)).

    Check:

    So, if the Cygwin .dll exposes some long value greater than 2 ** 64 (1 << 64), it will be truncated in the Win process, and in this case a crash might occur. Theoretically, this situation should affect the reverse scenario as well, but it doesn't.

    There are other factors that might lead to this behavior, like default memory alignment, and so on.