I know this has been discussed a few times, but my situation is a bit different.
I have a third-party dll exporting some classes. Unfortunately, the header file is not available. It is still possible to call exported functions. But I cannot get around passing the right 'this' pointer (which is passed in RCX register).
First I use dumpbin /exports to extract the function names (names are changed as the third-party library and function names are confidential).
4873 1308 0018B380 ?GetId@ThirdPartyClass@ThirdPartyNamespace@@QEBAJXZ = ??GetId@ThirdPartyClass@ThirdPartyNamespace@@QEBAJXZ (public: long __cdecl ThirdPartyNamespace::ThirdPartyClass::GetId(void)const )
Now, the API allows me to register my callback that receives a pointer to ThirdPartyNamespace::ThirdPartyClass (there is only forward declaration of ThirdPartyClass).
Here how I am trying to call ThirdPartyNamespace::ThirdPartyClass::GetId():
long (ThirdPartyNamespace::ThirdPartyClass::*_pFnGetId)() const;
HMODULE hModule = GetModuleHandle("ThirdPartyDLL.dll");
*(FARPROC*)&_pFnGetId= GetProcAddress(hModule, "?GetId@ThirdPartyClass@ThirdPartyNamespace@@QEBAJXZ");
long id = (ptr->*_pFnGetId)();
Everything looks fine (i.e. if I step in - I get indeed inside ThirdPartyClass::GetId method. But the this pointer is not good. While the ptr is good and if in debugger I manually change rcx to the ptr - it works fine. But compiler does not pass ptr for some reason. Here is disassembly:
long id = (ptr->*_pFnGetId)();
000000005C882362 movsxd rax,dword ptr [rdi+30h]
000000005C882366 test eax,eax
000000005C882368 jne MyClass::MyCallback+223h (05C882373h)
000000005C88236A movsxd rcx,dword ptr [rdi+28h]
000000005C88236E add rcx,rsi
000000005C882371 jmp MyClass::MyCallback+240h (05C882390h)
000000005C882373 movsxd r8,dword ptr [rdi+2Ch]
000000005C882377 mov rcx,rax
000000005C88237A mov rax,qword ptr [r8+rsi]
000000005C88237E movsxd rdx,dword ptr [rax+rcx]
000000005C882382 movsxd rcx,dword ptr [rdi+28h]
000000005C882386 lea rax,[r8+rdx]
000000005C88238A add rcx,rax
000000005C88238D add rcx,rsi
000000005C882390 call qword ptr [rdi+20h]
000000005C882393 mov ebp,eax
Before executing these commands, rsi contains the pointer to the object of ThirdPartyClass (i.e. ptr), but instead of passing it in rcx directly, some arithmetic is performed on it and as a result, this pointer gets completely wrong.
some traces which I don't understand why compiler is doing it as it end up calling non-virtual function ThirdPartyClass::GetId():
000000005C88237A mov rax,qword ptr [r8+rsi]
R8 0000000000000000
RSI 000000004C691AA0 // good pointer to ThirdPartyClass object
RAX 0000000008E87728 // this gets pointer to virtual functions table of ThirdPartyClass
000000005C88237E movsxd rdx,dword ptr [rax+rcx]
RAX 0000000008E87728
RCX FFFFFFFFFFFFFFFF
RDX FFFFFFFFC0F3C600
000000005C882382 movsxd rcx,dword ptr [rdi+28h]
RCX 0000000000000000
RDI 000000005C9BE690
000000005C882386 lea rax,[r8+rdx]
RAX FFFFFFFFC0F3C600
RDX FFFFFFFFC0F3C600
R8 0000000000000000
000000005C88238A add rcx,rax
RAX FFFFFFFFC0F3C600
RCX FFFFFFFFC0F3C600
000000005C88238D add rcx,rsi
RCX 000000000D5CE0A0
RSI 000000004C691AA0
000000005C882390 call qword ptr [rdi+20h]
In my view, it should be as simple as
long id = (ptr->*_pFnGetId)();
mov rcx,rsi
call qword ptr [rdi+20h]
mov ebp,eax
And if I set rcx equal to rsi before the call qword ptr [rdi+20h] it returns me expected value.
Am I doing something completely wrong? Thanks in advance.
Ok, I found a solution, by incident (as I already used similar approach and it worked in slightly different situation.
The solution is to trick the compiler by defining a fake class and calling member method by pointer, but pretending that it is a pointer to the known (to compiler) class.
Perhaps, it does not matter, but I know that ThirdPartyNamespace::ThirdPartyClass has virtual functions, so I declare fake class with virtual function as well.
class FakeCall
{
private:
FakeCall(){}
virtual ~FakeCall(){}
};
The rest as in the initial code except once small thing, instead of calling ptr->*_pFnGetId (where ptr is pointer to unknown, forward declared class ThirdPartyNamespace::ThirdPartyClass), I am pretending I am calling member method in my FakeCall class:
FakeCall * fake = (FakeCall*)ptr;
long sico = (fake->*_pFnGetId)();
Disassembly looks exactly as expected:
long sico = (fake->*_pFnGetSico)();
000000005A612096 mov rcx,rax
000000005A612099 call qword ptr [r12+20h]
000000005A61209E mov esi,eax
And it works perfectly!
Some observations: