windowspowershellpowershell-isepowershell-5.1

How to correct for PowerShell ISE string issue with external command on Windows 11


I have a PowerShell script (targeting 5.1) which when run using PowerShell ISE on Windows 10 works fine but in Windows 11 fails. The script pipes string literals to sqlite3.exe, for example

".show" | & .\sqlite3.exe db

Successful output on Windows 10:

    echo: off
     eqp: off
 explain: auto
 headers: off
    mode: list    nullvalue: ""
  output: stdout colseparator: "|" rowseparator: "\n"
   stats: off
   width:
filename: db

Output on Windows 11:

sqlite3.exe : Parse error near line 1: near ".": syntax error At
line:1 char:11
+ ".show" | & .\sqlite3.exe db
+ ~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (Parse error nea...": syntax error:String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError
.show
^--- error here

So it appears from the last 2 lines that ISE is adding what I assume is a byte-order mark to the beginning of the string which sqlite3.exe cannot handle.

I realise the obvious option is tell users to just not run it in ISE but I was wondering if there was any way of configuring ISE or rewriting the script to avoid the error?


Solution

  • tl;dr


    Indeed, the Windows PowerShell ISE is best avoided (to add to iRon's helpful comment on the question):

    Its poor support for calling external programs, such as sqlite3.exe, is one of the reasons to avoid it, and you've run into one aspect of it:


    Because $OutputEncoding is a UTF-8 encoding with BOM in the ISE by default, any data you send via the pipeline to an external program will have a BOM prepended - which most external programs do NOT expect, including sqlite3.exe, causing them to interpret the 3 bytes that comprise the BOM as part of the text.

    Because [Console]::OutputEncoding in the ISE is the active ANSI code page, and PowerShell uses it to decode stdout and stderr output from external programs, it mis-decodes the UTF-8 BOM contained in sqlite3.exe's stderr output before the word .show (sqlite3.exe assumes a fixed encoding of UTF-8, but doesn't recognize a BOM): The UTF-8 BOM is composed of 3 bytes, 0xef, 0xbb, 0xbf, which is mis-decoded as string  (i.e. 3 characters) if the active ANSI code page is Windows-1252, such as in US-English.[1]


    Solution options:

    Do one of the following before calling sqlite3.exe:


    [1] You can verify this as follows: [System.Text.Encoding]::GetEncoding(1252).GetString( [System.Text.UTF8Encoding]::new($true).GetPreamble())