c++rustclangvirtual-functionsvtable

Calling a C++ virtual method from Rust throws an Access violation error even after it executed successfully


I'm trying to call a virtual method of a C++ object from Rust. I'm getting output, but after the execution of this method, it throws an exception.

Unhandled exception at 0x00000001 in testvirtual.exe: 0xC0000005: Access violation executing location 0x00000001.

Here's a minimal repro of my issue

#include <iostream>
#include<Windows.h>

struct Calculate{
    virtual int sum(int a, int b) {
        return a + b;
    }
};

extern "C" __declspec(dllexport)  Calculate* get_numbers() {
    Calculate* obj = new Calculate();
    return obj;
}

int main() {
    typedef void(func)();
    auto handle = LoadLibrary(TEXT("testvirtualrs.dll"));
    auto address = (func*)GetProcAddress(handle, "execute");
    (*address)();
    return 0;
}

Rust:

#[link(name = "testvirtual")]
extern "C" {
    fn get_numbers() -> *mut Calculate;
}

#[repr(C)]
pub struct CalculateVtbl {
    pub sum: unsafe extern "C" fn(a: i32, b: i32) -> i32,
}

#[repr(C)]
pub struct Calculate {
    pub vtbl: *const CalculateVtbl,
}

impl Calculate {
    #[inline(always)]
    pub unsafe fn sum(&self, a: i32, b: i32) -> i32 {
        ((*self.vtbl).sum)(a, b)
    }
}

#[no_mangle]
pub unsafe extern "C" fn execute() {
    let a = get_numbers();
    unsafe {
       let val = (*a).sum(5, 6);
       println!("val = {val}");
    }
}

Output:

val = 11

Also I'm using clang compiler and x86 arch. Exception thrown after the method is finished calling from Rust, it was kind of ambiguous and it didn't help to find the cause


Solution

  • In C++, non-static methods are supposed to be called with a this pointer set. Nominally this is just a hidden first argument, but on x86 Windows there is a special thiscall calling convention that needs to be followed.

    You thus need to make every method in your vtable – as well as non-virtual methods – extern "thiscall":

    #[repr(C)]
    pub struct CalculateVtbl {
        pub sum: unsafe extern "thiscall" fn(this: *mut Calculate, a: i32, b: i32) -> i32,
    }
    
    #[repr(C)]
    pub struct Calculate {
        pub vtbl: *const CalculateVtbl,
    }
    
    impl Calculate {
        #[inline(always)]
        pub unsafe fn sum(&mut self, a: i32, b: i32) -> i32 {
            ((*self.vtbl).sum)(self, a, b)
        }
    }
    

    You can put this sample in Compiler Explorer and see what happens to ECX if you change thiscall back to C.