I'm writing small utility (VC 2010, no clr) that does one simple task (rasterizing) using 3rd party library. Later utility will be used by bigger application. Sometimes the utility crashes because of some heap corruption in 3rd party library. That is OK, but Windows (Vista/2008) shows well known dialog "Program has stopped working ... Close/Debug program." which is not appropriate in my case (Server side). Utility should crash/terminated silently w/o any visible effects.
To do that I installed SEH for unhandled exception (SetUnhandledExceptionFilter). The handler is perfectly invoked for exceptions like AV ( *(PDWORD)0 = 0 ), but for some reason it is not invoked in the case of heap corruption. Corruption happens in dllmain of one of 3rd party library dlls while it is being unloaded.
Couple of questions. Can anybody explain why the handler is not invoked? Is there any another way to prevent that dialog?
Apparently it's intentional that heap corruptions cannot be caught by user-defined exception handlers, even though they're emitted as exceptions with their own exception code (0xC0000374 "STATUS_HEAP_CORRUPTION"). Here's a Visual C++ bug report which was basically closed as "won't fix":
As you have discovered, this is not a bug in the compiler or the OS. The heap corruption that your function causes is treated as a critical error, and as part of handling that error the OS terminates the process. This is what causes your exception handlers to not be invoked.
I'd guess that Windows Error Reporting or other ways of creating a crash dump could still catch it.
As for preventing the dialog, in the registry you can either disable WER completely or just disable the dialog so that the process won't block:
https://msdn.microsoft.com/de-de/library/windows/desktop/aa366711(v=vs.85).aspx (see "DontShowUI")