I use WinSCP within a Powershell script. It suddenly stopped working. After a while I could figure out that the problem appeared from a more recent version of PowerShell:
Reduced code:
& winscp `
/log `
/command `
'echo Connecting...' `
"open sftp://kjhgk:jkgh@example.com/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""
Error message using v7.2.7
Host "example.com" does not exist.
Errror message using v7.3.0
Too many parameters for command 'open'.
As you can see with v7.3.0 WinSCP receives different input depending on the version of PS. I found out that the difference has something to do with the spaces in the hostkey. If they are omitted v7.3.0 outputs the same error.
What change to PowerShell caused this, and how can I fix it? (How can I debug such issues? I played a bit around with escaping, but the strings look the same no matter the version, no obvious breaking change that could be responsible)
Version 7.3.0 of PowerShell (Core) introduced a breaking change with respect to how arguments with embedded " characters (and empty-string arguments)[1] are passed to external programs, such as winscp:[2]
While this change is mostly beneficial, because it fixes behavior that was fundamentally broken since v1 (this answer discusses the old, broken behavior), it also invariably breaks existing workarounds that build on the broken behavior, except those for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js).
To make existing workarounds continue to work, set the $PSNativeCommandArgumentPassing preference variable (temporarily) to 'Legacy':
# Note: Enclosing the call in & { ... } makes it execute in a *child scope*
# limiting the change to $PSNativeCommandArgumentPassing to that scope.
& {
$PSNativeCommandArgumentPassing = 'Legacy'
& winscp `
/log `
/command `
'echo Connecting...' `
"open sftp://kjhgk:jkgh@example.com/ -hostkey=`"`"ssh-ed25519 includes spaces`"`""
}
Unfortunately, because winscp.exe only accepts
"open sftp://kjhgk:jkgh@example.com/ -hostkey=""ssh-ed25519 includes spaces""" on its process command line (i.e., embedded " escaped as ""), and not also the most widely used form
"open sftp://kjhgk:jkgh@example.com/ -hostkey=\"ssh-ed25519 includes spaces\"" (embedded " escaped as \"), which the fixed behavior now employs, for winscp.exe, specifically, a workaround will continue to be required.
If you don't want to rely on having to modify $PSNativeCommandArgumentPassing for the workaround, here are workarounds that function in both v7.2- and v7.3+ :
Use --%, the stop-parsing token, which, however, comes with pitfalls and severe limitations, notably the inability to (directly) use PowerShell variables or subexpressions in the arguments that follow it - see this answer for details; however, you can bypass these limitations if you use --% as part of an array that you construct and assign to a variable first and then pass via splatting:
# Note: Must be single-line; note the --% and the
# unescaped use of "" in the argument that follows it.
# Only "..." quoting must be used after --%
# and the only variables that can be used are cmd-style
# *environment variables* such as %OS%.
winscp /log /command 'echo Connecting...' --% "open sftp://kjhgk:jkgh@example.com/ -hostkey=""ssh-ed25519 includes spaces"""
# Superior alternative, using splatting:
$argList = '/log', '/command', 'echo Connecting...',
'--%', "open sftp://kjhgk:jkgh@example.com/ -hostkey=""ssh-ed25519 includes spaces"""
winscp @argList
Alternatively, call via cmd /c:
# Note: Pass-through command must be single-line,
# Only "..." quoting supported,
# and the embedded command must obey cmd.exe's syntax rules.
cmd /c @"
winscp /log /command "echo Connecting..." "open sftp://kjhgk:jkgh@example.com/ -hostkey=""ssh-ed25519 includes spaces"""
"@
@"<newline>...<newline>"@ or @'<newline>...<newline>'@), but it helps readability and simplifies using embedded quoting.Both workarounds allow you to pass arguments directly as quoted, but unfortunately also require formulating the entire (pass-through) command on a single line - except if --% is combined with splatting.
The v7.3+ default $PSNativeCommandArgumentPassing value on Windows, 'Windows':
regrettably retains the old, broken behavior for calls to batch files and the WSH CLIs (wscript.exe and cscript.exe) and their associated script files (with file-name extensions such as .vbs and .js), as well as find.exe and sqlcmd.exe (conceivably, more executables may be added in the future; see the docs for the current list).
While, for these programs only, this allows existing workarounds to continue to function, future code that only needs to run in v7.3+ will continue to be burdened by the need for these obscure workarounds, which build on broken behavior.
There are also troublesome signs that this list of exceptions will be appended to, piecemeal (which indeed happened for sqlcmd in v7.3.1, as well as temporarily for gcloud, which was later removed again), which all but guarantees confusion as to which programs require workarounds and which don't with respect to a given PowerShell version.
commendably, for all other programs PowerShell encodes the arguments - when it of necessity rebuilds the command line behind the scenes - as follows with respect to ":
It encodes the arguments for programs that follow the C++ command-line parsing rules (as used by C / C++ / .NET applications) / the parsing rules of the CommandLineToArgv WinAPI function, which are the most widely observed convention for parsing a process' command line.
In a nutshell, this means that embedded " characters embedded in an argument, to be seen as a verbatim part of it by the target program, are escaped as \", with \ itself requiring escaping only (as \\) if it precedes a " but is meant to be interpreted verbatim.
Note that if you set $PSNativeCommandArgumentPassing value to 'Standard' (which is the default on Unix-like platforms, where this mode fixes all problems and makes v7.3+ code never require workarounds), this behavior applies to all external programs, i.e. the above exceptions no longer apply).
For a summary of the impact of the breaking v7.3 change, see this comment on GitHub.
If you have / need to write cross-edition, cross-version PowerShell code: The Native module (Install-Module Native; authored by me), has an ie function (short for: Invoke Executable), which is a polyfill that provides workaround-free cross-edition (v3+), cross-platform, and cross-version behavior in the vast majority of cases - simply prepend ie to your external-program calls.
Caveat: In the specific case at hand it will not work, because it isn't aware that winscp.exe requires ""-escaping.
[1] See this answer for details and workarounds.
[2] Reverting that change in a later version and making the new behavior opt-in was briefly considered, but decided against - see GitHub issue #18694