I'm trying to separate a few libraries from my Python project, into a C++ submodule, that should improve the performance, using Cython. Now, I've managed to make the module an compile it, but now I want to make possible for it to be used in more platforms, not just my computer, and that's when I found that Github provides a way to do this through GHA, moreso, I found that there is cibuildwheel which helps to manage all this work, but I haven't been able to make it work. Seems like the problem might come from the fact that my CPP module requires 2 non-native libraries(libass, and libjsoncpp), and apparently the platforms GH use for the build are not exactly equal, hence I haven't been able to discover how to include them in the build.
setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import os
import sys
# Determine platform-specific library paths
if sys.platform == "win32":
# Windows
vcpkg_root = os.getenv('VCPKG_ROOT', 'C:\\vcpkg') # Set your vcpkg path
libass_head = os.path.join(vcpkg_root, 'installed', 'x64-windows', 'include')
libass_lib = os.path.join(vcpkg_root, 'packages', 'libass_x64-windows', 'lib', 'ass.lib')
json_lib = os.path.join(vcpkg_root, 'installed', 'x64-windows', 'lib', 'jsoncpp.lib')
json_head = os.path.join(vcpkg_root, 'installed', 'x64-windows', 'include')
else:
# Linux
libass_lib = "/usr/lib"
libass_head = "/usr/include/ass"
json_lib = "/usr/local/lib"
json_head = "/usr/include/"
ext_module = Extension(
"test_module",
sources=["test_module.pyx", "testmodule.cpp"],
language="c++",
extra_compile_args=["-std=c++11"],
libraries=["ass", "jsoncpp"],
library_dirs=[libass_lib, json_lib],
include_dirs=[libass_head, json_head],
)
setup(
name="test_module",
version='0.1.0',
ext_modules=cythonize([ext_module]),
)
testmodule.cpp
#include <iostream>
#include <fstream>
#include <regex>
#include <map>
#include <cstring>
#include <ass/ass.h>
#include <json/json.h>
using namespace std;
// Rest of the file
build_wheel.yml
name: Python application
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
permissions:
contents: read
jobs:
build:
name: Build wheel for ${{ matrix.python }}-${{ matrix.buildplat[1] }}
runs-on: ${{ matrix.buildplat[0] }}
strategy:
fail-fast: false
matrix:
buildplat:
- [ubuntu-20.04, manylinux_x86_64]
#- [ubuntu-20.04, manylinux_i686]
#- [windows-2019, win_amd64]
- [windows-2019, win32]
python: ["cp39", "cp310"]
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Install dependencies (Windows)
if: contains(matrix.buildplat[0], 'windows')
run: |
# Alternative Windows method using vcpkg
git clone https://github.com/microsoft/vcpkg.git C:\vcpkg
C:\vcpkg\bootstrap-vcpkg.bat
C:\vcpkg\vcpkg install libass
C:\vcpkg\vcpkg install jsoncpp
- name: Install dependencies (Linux)
if: contains(matrix.buildplat[0], 'ubuntu')
run: |
sudo apt-get update
sudo apt-get install -y libass-dev libjsoncpp-dev
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt # Install dependencies from requirements.txt
- name: Show DEBUG Linux
if: contains(matrix.buildplat[0], 'ubuntu')
run: |
whereis libass
whereis libjsoncpp
whereis ass
whereis jsoncpp
ls /usr/include/ass
ls /usr/include
- name: Show DEBUG Windows
if: contains(matrix.buildplat[0], 'windows')
run: |
ls C:\vcpkg\installed\x64-windows\include
ls C:\vcpkg\packages
ls C:\vcpkg\packages\libass_x64-windows
ls C:\vcpkg\packages\libass_x64-windows\lib
ls C:\vcpkg\packages\libass_x64-windows\include
- name: Build wheels
uses: pypa/cibuildwheel@v2.8.1
with:
output-dir: wheelhouse
env:
CIBW_BUILD: ${{ matrix.python }}-${{ matrix.buildplat[1] }}
CIBW_SKIP: "cp34-* cp35-*"
CIBW_BEFORE_BUILD: |
pip install setuptools==59.6.0 wheel Cython
CIBW_ENVIRONMENT_WINDOWS: >
INCLUDE="C:\vcpkg\installed\x64-windows\include"
LIB="C:\vcpkg\installed\x64-windows\lib"
CXXFLAGS="-IC:\vcpkg\installed\x64-windows\include"
LDFLAGS="-LC:\vcpkg\installed\x64-windows\lib"
CIBW_ENVIRONMENT_LINUX: >
CXXFLAGS="-I/usr/include"
LDFLAGS="-L/usr/lib"
- name: Show generated wheels
run: ls wheelhouse
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: built-wheels
path: ./wheelhouse/*.whl
Sorry if the setup.py and yml looks very messy, but I've been trying many things to understand what's happening inside of the instance doing the build.
In the case of my Linux builds, it's throwing this error, which seems to imply is not finding either the headers and/or the code of the libraries.
testmodule.cpp:6:10: fatal error: ass/ass.h: No such file or directory
6 | #include <ass/ass.h>
And in the case of the Windows builds, seems like I've managed to make them point to the right headers, but it's throwing LINK : fatal error LNK1181: cannot open input file 'ass.lib', which I understand means the lib files are not the right ones(?) I've tried to use the ones at installed and packages in vcpkg, but neither are working.
Does someone knows what might be wrong either on my setup.py or build_wheels.yml?
Ok, I think I finally found what my problem was, so let me detail it a little:
First note that I was installing the libraries outside the cibuildwheel, this is apparently not right, hence I had to add a few lines like these into the cibuildwheel step:
CIBW_BEFORE_ALL_WINDOWS: >
C:\vcpkg\bootstrap-vcpkg.bat &&
C:\vcpkg\vcpkg integrate install &&
C:\vcpkg\vcpkg install libass &&
C:\vcpkg\vcpkg install jsoncpp
CIBW_BEFORE_ALL_LINUX: >
yum update &&
yum install -y libass-devel jsoncpp-devel
Second, for the Windows build, note that I'm running on a win32 build, which means 32 bits version, Window's VCPKG is a little shitty at controlling which version of a library is installing, and because it was adding files in the x64 folder, I though I was running on a 64-bits version, that's why when importing the bins into the project, it didn't work, they were no the right ones for the version, so I changed my Windows version to win_amd64 and it worked, furthermore, adding a few extra lines to control which versión of the libs are getting installed would make this more of a correct answer.
TL;DR
YAML
- name: Build wheels
uses: pypa/cibuildwheel@v2.20.0
with:
output-dir: wheelhouse
env:
CIBW_BUILD: ${{ matrix.python }}-${{ matrix.buildplat[1] }}
CIBW_SKIP: "cp34-* cp35-*"
CIBW_BEFORE_BUILD: >
pip install setuptools==59.6.0 wheel Cython
CIBW_BEFORE_ALL_WINDOWS: >
C:\vcpkg\bootstrap-vcpkg.bat &&
C:\vcpkg\vcpkg integrate install &&
C:\vcpkg\vcpkg install libass &&
C:\vcpkg\vcpkg install jsoncpp
CIBW_BEFORE_ALL_LINUX: >
yum update &&
yum install -y libass-devel jsoncpp-devel
setup.py
if sys.platform == "win32":
# Windows
# Check if 32-bits or 64-bits | TODO
vcpkg_root = os.getenv('VCPKG_ROOT', 'C:\\vcpkg') # Set your vcpkg path
libs = [
os.path.join(vcpkg_root, 'installed', 'x64-windows', 'lib'),
os.path.join(vcpkg_root, 'installed', 'x64-windows', 'bin'),
]
includes = [
os.path.join(vcpkg_root, 'installed', 'x64-windows', 'include'),
os.path.join(vcpkg_root, 'installed', 'x64-windows', 'include')
]
else:
# Linux
libs = [
"/usr/lib",
]
includes = [
"/usr/include/ass",
"/usr/include/jsoncpp",
]