c++detours

Trying to do DetourAttach but cannot convert my lua function's datatype to LPVOID

So this is how my function actually looks like

DetourAttach(&(LPVOID&)lua_tolstring, (PBYTE)tostring);

lua_tolstring is const char* and LPVOID gives me this error.

typedef void* LPVOID
invalid type conversion

How can i make this work?

Solution

  • You don't have the right semantics for DetourAttach. The first argument is a pointer-to-pointer-to-function, which should be initialized to the original function being hooked. The second argument is a pointer-to-function containing your hook function.

    See This blog for examples.

    So you can't just pass the function. You have to initialize a variable, e.g.:

    // Declaration of LUA API function in header
const char*lua_tostring (lua_State *L, int index);
// Your hook function must have this signature to match
const char*my_tostring (lua_State *L, int index);
// Your variable
const char* (*Real_lua_tostring)(lua_State *L, int index) = lua_tostring;
// Make the call
DetourAttach(&(LPVOID&)Real_lua_tolstring, (PVOID)my_tostring);