I have written a global hook that hooks using SetWindowsHookEx the WH_GETMESSAGE, WH_CALLWNDPROC and WH_CALLWNDPROCRET.
The hook dll creates a new thread in the hooked process, which, among other things, checks the audio state of the process and calls IAudioSessionManager2::GetSessionEnumerator().
Now the interesting part, I had called UnhookWindowsHookEx() from the hook host AND during the time my dll's worker thread was running the call to IAudioSessionManager2::GetSessionEnumerator(). That call was in the same thread's call stack, where the DllMain with DLL_PROCESS_DETACH was invoked. I assume the reason was, that GetSessionEnumerator() invokes GetMessage() function somewhere and the latter is reentrant. Unfortunately I do not remember precisely, but I think I saw that in the call stack.
But there are multiple important things I wonder about and things that remain unclear. So here come my related questions:
Unfortunately I probably cannot experimentally solve these questions since I have had my hooking (and also unhooking) code running on multiple computers for months before such situation with DllMain occured during unhooking. Though for some reason it then occured with four different programs at once...
Also please, would someone with enough reputation like to merge the tags "reentrant" and "reentrancy"?
So thanks to Hans I now know regarding point (4) that DllMain DLL_PROCESS_DETACH will not be reentrant with hook procedures.
Regarding the thread that the hook created, my log files currently indicate that if DllMain DLL_PROCESS_DETACH is injected into the stack of this thread then that thread indeed will be terminated after DllMain exits and will NOT run to completion. That should answer points (2) and (3). The question itself implicitly answers the point (1).
But for solving the problem for that thread the hook created, I assume that the DllMain DLL_PROCESS_DETACH can be prevented by calling
GetModuleHandleEx
(
GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS,
(LPCTSTR)DllMain,
&hModule_thread
)
and before the thread termination calling
FreeLibraryAndExitThread(hModule_thread, 0)
So using GetModuleHandleEx should answer the point (7), which in turn makes all other points irrelevant. Of course I have to use some IPC to trigger the thread termination in the hooked processes.
The remaining interesting question is point (5), but it is just out of curiosity:
"When precisely the DllMain DLL_PROCESS_DETACH can be invoked in a thread - are there some specific Windows API functions that need to be called and which in turn may lead to DllMain DLL_PROCESS_DETACH call, or could it happen at any instruction?"