dllhookc++builder32bit-64bitsetwindowshookex

Bizarre hook behavior 32/64 bit


I'm using a local hook (WH_KEYBOARD) with ms word (OpusApp). Well, as far as I know a 32bit app with a 32bit DLL must work only with 32bit target applications. The weird thing is that the program only works with 64bits apps!!! That is it, only with 64bits APPS! For example, it works with IE 64 but not with IE 32! The app and dll are 32bit compiled with radstudio XE2, I confirmed the version into PE header. In 32bit OSs, the app and dll doesn´t work.

I found no solutions on net and see no starting point to solve this weird problem.

The DLL code:

// Exported functions

extern "C" __declspec(dllexport)bool __stdcall InstallMouseHook(unsigned long, void *);

extern "C" __declspec(dllexport)bool __stdcall RemoveMouseHook();

// Callback Procedure Declaration

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam);

// Global variables

HHOOK HookHandle;
HINSTANCE DllInstance;
typedef void (__stdcall *CALLIT)(int,WPARAM,LPARAM);
CALLIT callIt = NULL;

int WINAPI DllEntryPoint(HINSTANCE hinst, unsigned long reason, void*)
{
  DllInstance=hinst;
  return 1;
}

bool __stdcall InstallMouseHook(unsigned long pid, void *function)
{

     callIt = ( CALLIT ) function;

     if (function == NULL) {

        ShowMessage("function is null!");

     } else if (callIt == NULL) {

        ShowMessage("callIt is null!");

     }  

     HookHandle=SetWindowsHookEx(WH_KEYBOARD ,reinterpret_cast<HOOKPROC> (HookProc),DllInstance,pid);

    if (HookHandle==NULL)return false;

    else return true;

}

bool __stdcall  RemoveMouseHook()
{
  if(UnhookWindowsHookEx(HookHandle)==0)
  {
    return false;
  }
  else return true;
}

LRESULT CALLBACK HookProc(int code, WPARAM wParam, LPARAM lParam)
{
    if (code<0) {
        return CallNextHookEx(HookHandle,code,wParam,lParam);
    }

    if (callIt != NULL) {
        callIt(code,wParam,lParam);
    } else {
        ShowMessage("HookProc - no function to execute OR 32/64 bits problem!");
    }

  //Call the next hook in the chain
  return CallNextHookEx(HookHandle,code,wParam,lParam);
}

The EXE calling code:

void __fastcall TfrmMouseHook::btnHookAppDllClick(TObject *Sender)
{
    HWND hWindow;
    unsigned long pid;

    String s = "MouseHookDLL.dll";
    DllHandle=LoadLibrary(s.w_str());
    MOUSEHOOKFCT_2 InstHook=reinterpret_cast<MOUSEHOOKFCT_2> (GetProcAddress(DllHandle,"InstallMouseHook"));

    hWindow = FindWindow(ComboBox1->Text.w_str(),NULL);

    if (!hWindow) {
        msg("hWindow fail");
        return;
    }

    pid = GetWindowThreadProcessId(hWindow ,0);
    if (!pid) {
       msg("pid fail");
       return;
    }

    if(!InstHook(pid, (void *) callIt )) {
        msg("Unable to install  hook!");
    } else {
        msg(" #### hook INSTALLED! ####");
    }


}

CALLIT callIt(code,wParam,lParam) {
    frmMouseHook->msg("hook callit: code="+IntToStr(code) +" wparam="+IntToStr(wParam)+" lparam="+IntToStr(lParam) );
}



   Call IT is a function pointer to a hooker app function.

    Any ideas will be very wellcome!

Solution

  • What happens? Besides MSDN or anyone else says, there are some bug in XE6, compiling DLL in newer versions of IDE make this behavior disappears, in fact, the new DLL crash and hook nothing.

    As Remy noted, by test I passed a function pointer to the DLL, one wrong thing to do but, when added with the wrong thing done by Embarcadero, sort of functioned.

    By now, and I know people will get mad, I put both methods (the wrong and the correct hooks) in the same DLL and in my application and... get crazy... could hook into 32 and 64 bits app with only one DLL.

    Don't believe? Install XE6 and try!

    And Works in windows 10 as well in windows 7.