For cross-platform parity reasons, my GetOpenFileName() specifies no owner and I explicitly disable all toplevel windows myself. The problem is re-enabling. In order to re-enable these windows correctly, I need to re-enable them before the dialog closes.
In the case of the user choosing a file, this is no issue: I just check for CDN_FILEOK in the hook procedure. No issues, no messed-up focus.
How can I do the same, but for cancelling the dialog box?
I have tried WM_DESTROY in the hook procedure, but that runs after the dialog box has been hidden (too late). I have tried WM_COMMAND, but that doesn't seem to cover all cases. I'm not sure what other options I have.
I need to target Windows XP and newer for now; that also means no Common Item Dialogs. Thanks!
Alternative: if there was a way to do a callback-based GetOpenFileName() that returned control to my message loop, like on Mac OS X with beginSheetModalForWindow:, I could be able to specify a parent window and avoid this hack.
There is no CDN notification when the dialog is canceled. If the user presses the Cancel button, you could try intercepting the BN_CLICKED notification that it sends to the dialog, or even subclass the button itself. But if the user cancels the dialog through other means (clicking the red X, pressing ESC, etc), you will likely have to catch the WM_CLOSE message instead.