cwindowsstack-overflowstdcalllibffi

Closures with the STDCALL ABI on x86 Windows cause Stack Overflow exceptions with LibFFI 3.2.1


I am using the libffi library (version 3.2.1) to enable API interfacing from a scripting language, by allowing the scripter to create prototypes for invoking API functions or defining API-compatible callback functions.

On x64, only one calling convention is used on Windows; WIN64 in libffi. This works perfectly fine with closures in the library and the scripter can easily create callback functions that can be used in calls like EnumChildWindows(HWND, WNDENUMPROC, LPARAM)

On x86, however, invoking EnumChildWindows with a closure created by LibFFI with the STDCALL ABI causes a stack overflow exception to occur. As per reported issues, it is due to wrongly allocated arguments in the call to the trampoline function. (x86 stdcall (win32) incorrectly aligns stack arguments, win32 x86 stdcall closure: incorrectly restored stack after closure call).

I have tried applying the patches mentioned in those issue reports, however they do not seem to fix the issue for my test cases. I have another test case to determine if it is truly the argument alignment, by creating a callback function without any arguments; calling it from an external binary. This STDCALL function is invoked correctly and no issues occur, as no argument alignment is required.

In desperate state, I attempted to compile the tip version of the libffi repository with the same toolchain I used for 3.2.1, however that version complains that x86 windows targets are not supported (wrong asm syntax in sysv.S). I need the library to compile in MSVC, due to the static linking of CRT to the resulting static library.

Now in the end, my question for a solution consists of several options

I have been looking for solutions for over a week, and I'm assuming I'm missing something obvious as the library is used in many projects.

Thank you,
Bas


Solution

  • In further research I discovered that there is a fork of libffi used in Java Native Access. This version of LibFFI contains the fix I was looking for, and x86 stdcall closures are executed successfully. This version of LibFFI compiles with MSVCC successfully.

    This counts as the first solution in my list of possible options in my original question.