winapisetwindowshookexreentrancygetmessagedllmain

DllMain DLL_PROCESS_DETACH and GetMessage Function reentrancy


I have written a global hook that hooks using SetWindowsHookEx the WH_GETMESSAGE, WH_CALLWNDPROC and WH_CALLWNDPROCRET.
The hook dll creates a new thread in the hooked process, which, among other things, checks the audio state of the process and calls IAudioSessionManager2::GetSessionEnumerator().

Now the interesting part, I had called UnhookWindowsHookEx() from the hook host AND during the time my dll's worker thread was running the call to IAudioSessionManager2::GetSessionEnumerator(). That call was in the same thread's call stack, where the DllMain with DLL_PROCESS_DETACH was invoked. I assume the reason was, that GetSessionEnumerator() invokes GetMessage() function somewhere and the latter is reentrant. Unfortunately I do not remember precisely, but I think I saw that in the call stack.

But there are multiple important things I wonder about and things that remain unclear. So here come my related questions:

  1. Can DllMain with DLL_PROCESS_DETACH invoked any time, even in a thread which runs functions from that dll which is currently being unloaded?
  2. What happens with the functions up the stack when DllMain DLL_PROCESS_DETACH exits? Will the code in the functions up the call stack execute eventually?
  3. What if these functions do not exit? When will the dll be unloaded?
  4. Can the DllMain DLL_PROCESS_DETACH be similarly invoked during the callbacks for WH_GETMESSAGE, WH_CALLWNDPROC and WH_CALLWNDPROCRET hooks? I know and have experimentally confirmed that sometimes, although not too often, these functions are reentrant, so the calls to these functions can be injected during the time previous call is still running in the same stack, but I do not know whether also calls to DllMain can be injected in a similar manner.
  5. When precisely the DllMain can be invoked in a thread - are there some specific Windows API functions that need to be called and which in turn may lead to DllMain DLL_PROCESS_DETACH call, or could it happen at any instruction?
  6. If the DllMain DLL_PROCESS_DETACH call can be "injected" at any time AND functions up the call stack do not get executed anymore, then how do I know where precisely was the function up the call stack interrupted? So I could inside DllMain release some handles or resources allocated by the function up the stack.
  7. Is there any way to temporarily prevent/postpone calls to DllMain DLL_PROCESS_DETACH? Locks obviously do not help if the call/interruption occurs in the same stack.

Unfortunately I probably cannot experimentally solve these questions since I have had my hooking (and also unhooking) code running on multiple computers for months before such situation with DllMain occured during unhooking. Though for some reason it then occured with four different programs at once...


Also please, would someone with enough reputation like to merge the tags "reentrant" and "reentrancy"?


Solution

  • So thanks to Hans I now know regarding point (4) that DllMain DLL_PROCESS_DETACH will not be reentrant with hook procedures.

    Regarding the thread that the hook created, my log files currently indicate that if DllMain DLL_PROCESS_DETACH is injected into the stack of this thread then that thread indeed will be terminated after DllMain exits and will NOT run to completion. That should answer points (2) and (3). The question itself implicitly answers the point (1).

    But for solving the problem for that thread the hook created, I assume that the DllMain DLL_PROCESS_DETACH can be prevented by calling

    GetModuleHandleEx
    (
        GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS,
        (LPCTSTR)DllMain,
        &hModule_thread
    )
    

    and before the thread termination calling

    FreeLibraryAndExitThread(hModule_thread, 0)
    

    So using GetModuleHandleEx should answer the point (7), which in turn makes all other points irrelevant. Of course I have to use some IPC to trigger the thread termination in the hooked processes.

    The remaining interesting question is point (5), but it is just out of curiosity:
    "When precisely the DllMain DLL_PROCESS_DETACH can be invoked in a thread - are there some specific Windows API functions that need to be called and which in turn may lead to DllMain DLL_PROCESS_DETACH call, or could it happen at any instruction?"