pythoncmakecross-compilingswigdllimport

ImportError: DLL load failed: The specified module could not be found for custom built pkg using Swig & CMake

I want to make a Python Package for win_amd64 (Windows x64) of cpp files. I have used Swig to wrap them into python and then used setuptools to build it. I have made CMakeLists.txt files to do all this actions. Note: I am using x86_64-w64-mingw32 cross-compiler to build this in linux machine

Tools Used & its Version:

Note: I can build the python pkg for linux-x86_64 (linux 64) for all of Python3.6 and above, but for windows I can only build for Python3.6 (By using the library libpython36.a) but with later version for win_amd64 I had to use the libpython39.lib (Lib based libraries)

What happens now:

  1. I can build the whl file for win_amd64 successfully. CMake can make the .dll and setuptools wrapped it well

  2. On installing this on windows Python3.9 x64, it installs without any warning

  3. But when I try to import the main module, it throws

    import sample # name of the module
# Error below
Traceback:
from . import _sample_swig
ImportError: DLL load failed: The specified module could not be found.ImportError: DLL load failed: The specified module could not be found.`

Project Setup:

  1. Swig File

    FIND_PACKAGE(SWIG REQUIRED)
INCLUDE(${SWIG_USE_FILE})
CMAKE_MINIMUM_REQUIRED(VERSION 3.16)
SET(PYTHON_MODULE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/sample)
set(PYTHON_EXECUTABLE python${PYTHON_BUILD_VERSION})
if (${BUILD_PLATFORM} MATCHES "Windows")
    set(CMAKE_SYSTEM_NAME Windows)
    set(CMAKE_C_COMPILER   /usr/bin/x86_64-w64-mingw32-gcc)
    set(CMAKE_CXX_COMPILER /usr/bin/x86_64-w64-mingw32-g++)
    set(PYTHON_LIBRARY /home/developer/workspace/69/py39/libs/python.lib)
    set(PYTHON_INCLUDE_DIR /home/developer/workspace/69/py39/include)
    set(TARGET_EXTENSION "pyd")
    set(CMAKE_FIND_ROOT_PATH /opt/win_cross_compile_libs)

ELSE()
    set(CMAKE_C_COMPILER /usr/bin/gcc)
    set(CMAKE_CXX_COMPILER /usr/bin/g++)
    set(TARGET_EXTENSION "so")
    set(PythonLibs_DIR /usr/lib/x86_64-linux-gnu/)
    set(CMAKE_FIND_ROOT_PATH /usr/lib/x86_64-linux-gnu/)
ENDIF()
project(sample LANGUAGES CXX)
set(CMAKE_LANGUAGE_COMPILER ${CMAKE_CXX_COMPILER})
FIND_PACKAGE(Python3 COMPONENTS Interpreter Development)
FIND_PACKAGE(PythonLibs)
INCLUDE_DIRECTORIES(${PYTHON_INCLUDE_DIRS} ${PROJECT_SOURCE_DIR}/include)
message(STATUS "Py Libs: ${PYTHON_LIBRARIES}")
message(STATUS "Add. Linking Flags: ${ADD_LINK_FLAGS}")
message(STATUS "Python Incl Dir: ${PYTHON_INCLUDE_DIR}")
message(STATUS "Python Module: ${PYTHON_MODULE_DIR}")
message(STATUS "Machine: ${CMAKE_SYSTEM_NAME}")
set(PythonLibs_DIR "/usr/lib/x86_64-linux-gnu/")
set_property(SOURCE include/sample.i PROPERTY CPLUSPLUS ON)
file(GLOB SRC_FILES ${PROJECT_SOURCE_DIR}/src/*.cpp)
swig_add_library(sample_swig TYPE SHARED
          LANGUAGE python
          OUTPUT_DIR ${PYTHON_MODULE_DIR}
          OUTFILE_DIR src
          SOURCES include/sample.i ${SRC_FILES}
          )
SWIG_LINK_LIBRARIES(sample_swig ${PYTHON_LIBRARIES})
add_custom_command(
TARGET _sample_swig POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy
     _sample_swig.${TARGET_EXTENSION}
     ${PYTHON_MODULE_DIR}/_sample_swig.${TARGET_EXTENSION}
)

INSTALL(TARGETS _sample_swig LIBRARY DESTINATION sample)

  2. Interface file used for swig:

    %module arithmetic_operator
%{
    #include "addition.h"
%}
%include "addition.h"
%include "stdint.i"`

  3. Header file:

    # include <iostream>
using namespace std;

int addition_fn(int a, int b)
{
    return (a+b);`
}

  4. CPP File:

    # include <iostream>
using namespace std;
const int MAX_ADDABLE = 50;
int main()
{
    cout<<"Hello World\n";
}

  5. Setup.py:

    from skbuild import setup

setup(
name='sample',
version='0.1.0',
description='Sample',
cmake_with_sdist=True,
packages=['sample'],
python_requires=">=3.6",
include_package_data=True,
zip_safe=False
)

Using os.add_dll_directory(<dll folder location (site-packages/sample)>) as the Python3.8 and above only loads from trusted sources even though overwritten by PYTHONPATH - Doesn't work

Solution

  • Wasn't able to figure it out for a month.

    I finally fixed this by moving my x86_64-w64-mingw32-g++ which was on 9.0.x on Ubuntu 20.04 to compiler x86_64-w64-mingw32-g++-posix with GCC version 10-posix 20220113 (GCC) on Ubuntu-22.04

    Seems like the latest GCC will seemingly handle Windows Python 3.8+ DLL based handles when using Shared Libraries