Detecting if stdout is a console with MS Visual compilation, with console provided by mingw64

I maintain a command line utility that generates binary data. Data can be redirected towards stdout when requested. This is valid when stdout is redirected to a pipe or a file, but less so when stdout is a console, as it would garbage the console.

In order to protect users against such mistake, the program must detect if stdout is a console or not, and bail out when it is.

Now, this is nothing new, and a quick look over Internet will find multiple solutions. The main drawback is that there is no "universal" method, and Visual Studio requires its own flavor.

The console detector I'm using for Visual has a flaw : it doesn't detect that stdout is a console when the console is provided by mingw, which I believe means that it is mintty.

Here is the relevant code section :

#if defined(WIN32) || defined(_WIN32)
#  include <io.h>      /* _isatty */
#  include <windows.h> /* DeviceIoControl, HANDLE, FSCTL_SET_SPARSE */
#  include <stdio.h>   /* FILE */
static __inline int IS_CONSOLE(FILE* stdStream) {
    DWORD dummy;
    return _isatty(_fileno(stdStream)) && 
           GetConsoleMode((HANDLE)_get_osfhandle(_fileno(stdStream)), &dummy);
}
#endif

Note that the console detector works fine with the built-in Windows console (conhost.exe). It also works fine when the binary is compiled by mingw64. So the issue is mostly "compiled with Visual + console is mintty".

I've been looking around for some potential backup solutions, and found multiple variants of console detector for Visual, using different logics. But none of them would identify mintty as a console, they all fail.

I'm wondering if it is a problem with mintty, though I would anticipate that if it is, more applications would be impacted. Yet, searching for such issue over Internet yields relatively little complaints, and no solution.

Is it a known issue ?
Is there a known solution ?

Solution

mintty is a terminal emulator and does not present a console to a running application. When I need to run a true console program I've got to use winpty. For example winpty powershell will allow powershell to run correctly within mintty.

It is a known issue that several applications like git works around. This is what I also found.

https://github.com/fusesource/jansi-native/issues/11.

https://github.com/fusesource/jansi-native/commit/461068c67a38647d2890e96250636fc0117074f5

So apparently you should also check if you are connected to a pipe with the following name:

/*
 * Check if this could be a MSYS2 pty pipe ('msys-XXXX-ptyN-XX')
 * or a cygwin pty pipe ('cygwin-XXXX-ptyN-XX')
 */