I'm loading a not-owned-by-me library with Python's ctypes module as ctypes.CDLL("libthirdparty.so") which produces an error undefined symbol: g_main_context_push_thread_default because libthirdparty.so was overlinking a lot of unneeded / unused stuff like glib.
In this particular case, I can work around this problem successfully by LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0 python ...-ing glib, but I'm curious in two generic related questions touching on how ctypes works:
Is it possible to ask ctypes / dlopen to ignore undefined symbols, given that I know that these undefined symbols are not used on the code path I'm interested in (or otherwise I would be okay if it crashed in a hard way)?
Is it possible to ask ctypes to load several libraries at once or emulate LD_PRELOAD-ing? I tried to do ctypes.CDLL("/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0") in attempt to force-load glib in the Python's process before loading ctypes.CDLL("libthirdparty.so"), but it did not help.
Thanks!
Listing [Python.Docs]: ctypes - A foreign function library for Python.
What you're after, is [Man7]: DLOPEN (3) (emphasis is mine):
RTLD_LAZY
Perform lazy binding. Resolve symbols only as the code that references them is executed. If the symbol is never referenced, then it is never resolved. (Lazy binding is performed only for function references; references to variables are always immediately bound when the shared object is loaded). Since glibc 2.1.1, this flag is overridden by the effect of the LD_BIND_NOW environment variable.
Here's an example:
common.h:
#pragma once #include <stdio.h> #define C_TAG "From C" #define PRINT_MSG_0() printf("%s - [%s] (%d) - [%s]\n", C_TAG, __FILE__, __LINE__, __FUNCTION__)
dll01.h:
#pragma once #if defined(_WIN32) # define DLL01_EXPORT_API __declspec(dllexport) #else # define DLL01_EXPORT_API #endif #if defined(__cplusplus) extern "C" { #endif DLL01_EXPORT_API int inner(); #if defined(__cplusplus) } #endif
dll01.c:
#include "dll01.h" #include "common.h" # if !defined(NO_EXPORTS) int inner() { PRINT_MSG_0(); return 0; } #endif
dll00.c:
#include <stdio.h> #include "common.h" #include "dll01.h" #if defined(_WIN32) # define DLL00_EXPORT_API __declspec(dllexport) #else # define DLL00_EXPORT_API #endif #if defined(__cplusplus) extern "C" { #endif DLL00_EXPORT_API int simple(); DLL00_EXPORT_API int innerWrapper(); #if defined(__cplusplus) } #endif int simple() { PRINT_MSG_0(); return 0; } int innerWrapper() { PRINT_MSG_0(); return inner(); }
code00.py:
#!/usr/bin/env python import ctypes as cts import os import sys DLL_NAME = "./libdll00.{:s}".format("dll" if sys.platform[:3].lower() == "win" else "so") def main(*argv): mode = cts.DEFAULT_MODE for arg in argv: flag = getattr(os, f"RTLD_{arg.upper()}", None) if flag is not None: mode |= flag print(f"Loading .dll (mode: {mode})...") dll = cts.CDLL(DLL_NAME, mode=mode) for func_name in ("simple", "innerWrapper"): print(f"Loading {func_name} function...") func = getattr(dll, func_name) func.argtypes = () func.restype = cts.c_int res = func() print(f"{func_name} returned: {res}") if __name__ == "__main__": print( "Python {:s} {:03d}bit on {:s}\n".format( " ".join(elem.strip() for elem in sys.version.split("\n")), 64 if sys.maxsize > 0x100000000 else 32, sys.platform, ) ) rc = main(*sys.argv[1:]) print("\nDone.\n") sys.exit(rc)
Output:
(qaic-env) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackExchange/StackOverflow/q078879247]> ~/sopr.sh ### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ### [064bit prompt]> ls code00.py common.h dll00.c dll01.c dll01.h [064bit prompt]> [064bit prompt]> gcc -fPIC -shared dll01.c -o libdll01.so [064bit prompt]> LIBRARY_PATH="${LIBRARY_PATH}:." gcc -fPIC -shared dll00.c -o libdll00.so -ldll01 [064bit prompt]> ls code00.py common.h dll00.c dll01.c dll01.h libdll00.so libdll01.so [064bit prompt]> export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:." [064bit prompt]> ldd libdll00.so linux-vdso.so.1 (0x00007ffef55a7000) libdll01.so => ./libdll01.so (0x00007f032db60000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f032d94c000) /lib64/ld-linux-x86-64.so.2 (0x00007f032db6c000) [064bit prompt]> [064bit prompt]> # ----- Normal libdll01.so [064bit prompt]> # --- Symbol resolution: dlopen [064bit prompt]> python ./code00.py Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux Loading .dll (mode: 0)... Loading simple function... From C - [dll00.c] (26) - [simple] simple returned: 0 Loading innerWrapper function... From C - [dll00.c] (33) - [innerWrapper] From C - [dll01.c] (8) - [inner] innerWrapper returned: 0 Done. [064bit prompt]> # --- Symbol resolution: lazy [064bit prompt]> python ./code00.py lazy Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux Loading .dll (mode: 1)... Loading simple function... From C - [dll00.c] (26) - [simple] simple returned: 0 Loading innerWrapper function... From C - [dll00.c] (33) - [innerWrapper] From C - [dll01.c] (8) - [inner] innerWrapper returned: 0 Done. [064bit prompt]> [064bit prompt]> gcc -fPIC -shared dll01.c -o libdll01.so -DNO_EXPORTS [064bit prompt]> # ----- Messed up (undefined "inner" function) libdll01.so [064bit prompt]> # --- Symbol resolution: dlopen [064bit prompt]> python ./code00.py Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux Loading .dll (mode: 0)... Traceback (most recent call last): File "./code00.py", line 41, in <module> rc = main(*sys.argv[1:]) File "./code00.py", line 18, in main dll = cts.CDLL(DLL_NAME, mode=mode) File "/usr/lib/python3.8/ctypes/__init__.py", line 373, in __init__ self._handle = _dlopen(self._name, mode) OSError: ./libdll00.so: undefined symbol: inner [064bit prompt]> [064bit prompt]> # --- Symbol resolution: lazy [064bit prompt]> python ./code00.py lazy Python 3.8.10 (default, Jul 29 2024, 17:02:10) [GCC 9.4.0] 064bit on linux Loading .dll (mode: 1)... Loading simple function... From C - [dll00.c] (26) - [simple] simple returned: 0 Loading innerWrapper function... From C - [dll00.c] (33) - [innerWrapper] python: symbol lookup error: ./libdll00.so: undefined symbol: inner
Might also worth reading:
[SO]: C function called from Python via ctypes returns incorrect value (@CristiFati's answer)
[SO]: Resolving circular shared-object dependencies with ctypes/cffi (@CristiFati's answer)
Regarding your 2nd question, loading the library beforehand has no effect cause it can be found by the loader anyway, so no matter if loaded manually before or automatically by libthirdparty.so it's still the (same) wrong one.