WinAPI Hooking CreateFileW in Notepad

I made a DLL which should hook CreateFileW from Notepad but however its crashing. After debugging it I found out that its causing a stack overflow in the first line of the HookedCreateFile function:

(its saying that its causing a exception error at the address...)

The callstack at exception point:

My code:

typedef HANDLE(WINAPI * CreateFileFn)(
    LPCWSTR lpFileName,
    DWORD dwDesiredAccess,
    DWORD dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD dwCreationDisposition,
    DWORD dwFlagsAndAttributes,
    HANDLE hTemplateFile);

CreateFileFn oCreateFile = (CreateFileFn)GetProcAddress(GetModuleHandleA("kernel32.dll"), "CreateFileW");

HANDLE WINAPI HookedCreateFile(
    LPCWSTR lpFileName,
    DWORD dwDesiredAccess,
    DWORD dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD dwCreationDisposition,
    DWORD dwFlagsAndAttributes,
    HANDLE hTemplateFile)
{
    //std::cout << "Hello!" << std::endl;

    return oCreateFile(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes, dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);
}

I am using DetourFunction from the Microsoft Detours:

DetourFunction((PBYTE)oCreateFile, (PBYTE)HookedCreateFile);

First, DetourFunction() is old, it has been replaced with DetourAttach(). You should update your code to use a new version of the Detours library. See Microsoft's wiki on Using Detours.

Second, when you detour a function, you are replacing the first few instructions of the function with a jump into your hook function. DetourFunction() returns a trampoline, which you MUST use to call the original function. The trampoline executes the instructions that were replaced, and then jumps to the remaining unhooked code of the original function.

But, your hook is not using the trampoline at all, so every time it calls oCreateFile, it ends up calling back into itself, over and over in an endless recursive loop. That is what is causing the stack overflow error, as every call pushes another copy of the input parameters onto the call stack. Eventually, the call stack runs out of available space.

Try this instead:

CreateFileFn origCreateFile = (CreateFileFn) GetProcAddress(GetModuleHandle(TEXT("kernel32.dll")), "CreateFileW");
CreateFileFn trampCreateFile;

HANDLE WINAPI HookedCreateFile(
    LPCWSTR lpFileName,
    DWORD dwDesiredAccess,
    DWORD dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD dwCreationDisposition,
    DWORD dwFlagsAndAttributes,
    HANDLE hTemplateFile)
{
    //std::cout << "Hello!" << std::endl;
    return trampCreateFile(lpFileName, dwDesiredAccess, dwShareMode, lpSecurityAttributes, dwCreationDisposition, dwFlagsAndAttributes, hTemplateFile);
}

...

trampCreateFile = (CreateFileFn) DetourFunction((PBYTE)origCreateFile, (PBYTE)HookedCreateFile);