I have a PowerShell script (targeting 5.1) which when run using PowerShell ISE on Windows 10 works fine but in Windows 11 fails. The script pipes string literals to sqlite3.exe, for example
".show" | & .\sqlite3.exe db
Successful output on Windows 10:
echo: off eqp: off explain: auto headers: off mode: list nullvalue: "" output: stdout colseparator: "|" rowseparator: "\n" stats: off width: filename: db
Output on Windows 11:
sqlite3.exe : Parse error near line 1: near ".": syntax error At
line:1 char:11
+ ".show" | & .\sqlite3.exe db
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Parse error nea...": syntax error:String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError
.show
^--- error here
So it appears from the last 2 lines that ISE is adding what I assume is a byte-order mark to the beginning of the string which sqlite3.exe cannot handle.
I realise the obvious option is tell users to just not run it in ISE but I was wondering if there was any way of configuring ISE or rewriting the script to avoid the error?
tl;dr
Either: Bypass the problem by passing .show
as an argument (rather than via the pipeline) to sqlite3.exe
, as iRon suggests (note that I'm omitting &
, the caller operator, as it isn't necessary here):
.\sqlite3.exe db .show
Or: Fix the character-encoding problem, in the simplest case as follows, but note that this only works for ASCII-characters-only text:
$OutputEncoding = [Console]::OutputEncoding
'.show' | .\sqlite3.exe db
Indeed, the Windows PowerShell ISE is best avoided (to add to iRon's helpful comment on the question):
Its poor support for calling external programs, such as sqlite3.exe
, is one of the reasons to avoid it, and you've run into one aspect of it:
Inexplicably, the $OutputEncoding
preference variable defaults to UTF-8 in Windows 11; in Windows 10, it was - more sensibly - the system's active ANSI encoding, in sync with [Console]::OutputEncoding
.
In a regular Windows PowerShell console, it defaults to ASCII(!), which is equally mystifying. In PowerShell 7, it defaults to BOM-less UTF-8, which - while better due to not using a BOM - is also mystifying: see next point.
Given that console windows (as well as Windows Terminal tabs and Visual Studio Code's integrated terminal (I won't mention these explicitly below), but not the console emulation built into the ISE) default to the system's active legacy OEM code page, whose encoding is reflected in [Console]::OutputEncoding
(which PowerShell uses to decode external-program output), it would make sense to default $OutputEncoding
to the same code page (encoding).
Also inexplicably, the ISE defaults to the ANSI code page instead, resulting in different default behavior between it and console windows.
chcp
to verify), so that any non-ASCII characters in an external program's output will be misinterpreted by PowerShell in the ISE.What's worse, $OutputEncoding
in the ISE is - inappropriately - a UTF-8 encoding with BOM, which is what causes your problem - see below.
Because $OutputEncoding
is a UTF-8 encoding with BOM in the ISE by default, any data you send via the pipeline to an external program will have a BOM prepended - which most external programs do NOT expect, including sqlite3.exe
, causing them to interpret the 3 bytes that comprise the BOM as part of the text.
Because [Console]::OutputEncoding
in the ISE is the active ANSI code page, and PowerShell uses it to decode stdout and stderr output from external programs, it mis-decodes the UTF-8 BOM contained in sqlite3.exe
's stderr output before the word .show
(sqlite3.exe
assumes a fixed encoding of UTF-8, but doesn't recognize a BOM): The UTF-8 BOM is composed of 3 bytes, 0xef
, 0xbb
, 0xbf
, which is mis-decoded as string 
(i.e. 3 characters) if the active ANSI code page is Windows-1252, such as in US-English.[1]
Solution options:
Do one of the following before calling sqlite3.exe
:
Align $OutputEncoding
with [Console]::OutputEncoding
, which sends text in the same ANSI encoding that well-behaved external console programs expect - however, due to the mismatch between the ANSI encoding the ISE uses vs. the OEM encoding used by external programs, this solution is effective only if the input text as well as the external program's output is composed of ASCII-range characters only or if the target program happens to use the active ANSI code page, as python
does, for instance. (This in effect restores the Windows 10 behavior.)
# Effective only for all-ASCII-characters input and output text.
$OutputEncoding = [Console]::OutputEncoding
Switch the session to using BOM-less UTF-8 consistently, which enables full Unicode support on both the input and output side:
chcp >$null # dummy command that forces allocation of a console
$OutputEncoding = [Console]::InputEncoding = [Console]::OutputEncoding = [System.Text.UTF8Encoding]::new()
Note the need to execute a dummy command that calls a console application (chcp
) in order to ensure that the ISE has allocated a console behind the scenes; without that, assigning to [Console]::InputEncoding
or [Console]::OutputEncoding
would cause an error.
(Also, using chcp 65001
directly to change the active code page to UTF-8 would not work, because .NET caches the encodings stored in [Console]::InputEncoding
and [Console]::OutputEncoding
.)
See this answer for background information.
[1] You can verify this as follows: [System.Text.Encoding]::GetEncoding(1252).GetString( [System.Text.UTF8Encoding]::new($true).GetPreamble())