The following list does not sort properly (IMHO):
$a = @( 'ABCZ', 'ABC_', 'ABCA' )
$a | sort
ABC_
ABCA
ABCZ
My handy ASCII chart and Unicode C0 Controls and Basic Latin chart have the underscore (low line) with an ordinal of 95 (U+005F). This is a higher number than the capital letters A-Z. Sort should have put the string ending with an underscore last.
Get-Culture is en-US
The next set of commands does what I expect:
$a = @( 'ABCZ', 'ABC_', 'ABCA' )
[System.Collections.ArrayList] $al = $a
$al.Sort( [System.StringComparer]::Ordinal )
$al
ABCA
ABCZ
ABC_
Now I create an ANSI encoded file containing those same 3 strings:
Get-Content -Encoding Byte data.txt
65 66 67 90 13 10 65 66 67 95 13 10 65 66 67 65 13 10
$a = Get-Content data.txt
[System.Collections.ArrayList] $al = $a
$al.Sort( [System.StringComparer]::Ordinal )
$al
ABC_
ABCA
ABCZ
Once more the string containing the underscore/lowline is not sorted correctly. What am I missing?
Edit:
Let's reference this example #4:
'A' -lt '_'
False
[char] 'A' -lt [char] '_'
True
Seems like both statements should be False or both should be True. I'm comparing strings in the first statement, and then comparing the Char type. A string is merely a collection of Char types so I think the two comparison operations should be equivalent.
And now for example #5:
Get-Content -Encoding Byte data.txt
65 66 67 90 13 10 65 66 67 95 13 10 65 66 67 65 13 10
$a = Get-Content data.txt
$b = @( 'ABCZ', 'ABC_', 'ABCA' )
$a[0] -eq $b[0]; $a[1] -eq $b[1]; $a[2] -eq $b[2];
True
True
True
[System.Collections.ArrayList] $al = $a
[System.Collections.ArrayList] $bl = $b
$al[0] -eq $bl[0]; $al[1] -eq $bl[1]; $al[2] -eq $bl[2];
True
True
True
$al.Sort( [System.StringComparer]::Ordinal )
$bl.Sort( [System.StringComparer]::Ordinal )
$al
ABC_
ABCA
ABCZ
$bl
ABCA
ABCZ
ABC_
The two ArrayList contain the same strings, but are sorted differently. Why?
Many moons later, let me attempt a comprehensive summary:
By design:
PowerShell's Sort-Object does not perform ordinal sorting of strings (the latter being based strictly on the Unicode code points (sometimes still loosely, but incorrectly referred to as "ASCII values") of the characters in the string).
Instead, like the underlying .NET runtime, it performs culture-specific sorting based on linguistic rules by default.
Note: In PowerShell (Core) v7.1+ / .NET 5+ the sorting rules changed, due to the latter's move to the ICU libraries even on Windows (which had previously used NLS) - see .NET globalization and ICU
_ before ., spurred complaints, because it changes long-established behavior in the context of file-system-related operations - see GitHub issue #14757.Also, a notable difference is that PowerShell, unlike .NET, is generally case-insensitive by default, requiring the -CaseSensitive switch as an opt-in to case-sensitivity.
In other words: Sort-Object uses [StringComparer]::CurrentCultureIgnoreCase by default, and -CaseSensitive changes that to [StringComparer]::CurrentCulture
Sort-Object's -Culture parameter allows you to change the cultural context ad hoc; use -Culture 127 to choose the invariant culture (which PowerShell uses by default in many contexts - see this answer).
Unfortunately, as of PowerShell (Core) 7.4.x, unlike with direct use of .NET APIs, you can NOT make Sort-Object perform ordinal sorting.
Sort-Object to be enhanced so as to support ordinal sorting on demand, along with an associated RFC, https://github.com/PowerShell/PowerShell-RFC/pull/167. Unfortunately, activity has stalled quite a while ago.Unexpectedly:
As for why sorting the array resulting from $a = Get-Content data.txt vs. an array literal behaves differently - user4003407's helpful answer explains the technical underpinnings in detail, but let me attempt a pragmatic summary:
Behind the scenes, PowerShell situationally uses [psobject] wrappers around .NET objects, which are meant to be invisible helper objects, but, unfortunately, they aren't always invisible and sometimes cause unexpected behavior.
Notably, objects output from cmdlets (such as Get-Content) are invariably [psobject]-wrapped.
Thus, when such [psobject]-wrapped objects are implicitly collected in a regular PowerShell array - as happens when you capture the lines of a text file being streamed by Get-Content in a variable, for instance - you'll end up with an [object[]-typed array whose elements happen to be [psobject] instances (which happen to wrap [string] instances).
By contrast, if you capture an array of string literals in a variable - e.g. $a = @( 'ABCZ', 'ABC_', 'ABCA' ) - the resulting [object[]]-typed array has [string] elements, i.e. no [psobject] wrappers are involved.
When an [object[]]-typed array or non-generic array-like type such as System.Collections.ArrayList is used, the above distinction matters:
Only if the original array's elements are [string]-typed does a sorting qualifier such as [System.StringComparer]::Ordinal matter; otherwise, this qualifier is ignored, and comparison for sorting is delegated to whatever type the elements happen to be:
[psobject] happens to implement the IComparable interface itself, and it effectively delegates to the [string] type's implementation behind the scenes, albeit using the latter's default behavior - which is the culture-sensitive, case-sensitive sorting ([StringComparer]::CurrentCulture)).
This behavior is counter-intuitive, but cannot be changed for fear of breaking backward compatibility - see GitHub issue #14829 for discussion.
For a given array (list) $a, a simple test as to whether an element is [psobject]-wrapped is $a[0] -is [psobject], for instance.
To provide simplified examples:
# == OK: Elements are NOT [psobject]-wrapped
$a = '_', 'A'
[Array]::Sort($a, [StringComparer]::Ordinal)
$a # -> 'A', '_' - ORDINAL sorting
# (Code point of 'A' is 0x41 (65), of '_' is 0x5f (95))
[System.Collections.ArrayList] $al = '_', 'A'
$al.Sort([StringComparer]::Ordinal)
$al # -> 'A', '_' - ORDINAL sorting
# == !! BROKEN: Elements ARE [psobject]-wrapped, therefore
# !! [StringComparer]::Ordinal is *ignored*
# !! Using Write-Output causes the array elements to be [psobject]-wrapped,
# !! as Get-Content does.
$a = Write-Output '_', 'A'
# !! [StringComparer]::Ordinal is IGNORED, due to [psobject] wrappers.
[Array]::Sort($a, [StringComparer]::Ordinal)
$a # -> 'A', '_' - quiet fallback to the default, [StringComparer]::CurrentCulture
# !! Using Write-Output causes the array elements to be [psobject]-wrapped
[System.Collections.ArrayList] $al = Write-Output '_', 'A'
# !! [StringComparer]::Ordinal is IGNORED, due to [psobject] wrappers.
$al.Sort([StringComparer]::Ordinal)
$al # -> 'A', '_' - quiet fallback to the default, [StringComparer]::CurrentCulture
Workaround:
Casting an [object[]] array whose elements are [psobject]-wrapped to a specific type - such as [string[]] in the case at hand - implicitly discards the [psobject] wrappers:
# The [string[]] cast implicitly discards the [psobject] wrappers
# in the process of creating a *strongly typed* array.
[string[]] $a = Write-Output '_', 'A'
[Array]::Sort($a, [StringComparer]::Ordinal)
$a # -> 'A', '_' - ORDINAL sorting, as requested.
As for PowerShell's comparison operators and the inconsistent [char] handling:
PowerShell's comparison operators such as -eq and -lt:
With [string] instances, like Sort-Object, comparison operators use linguistic rules, however based on the invariant culture rather than on the current culture (the latter being reflected in [cultureinfo]::CurrentCulture / Get-Culture, $PSCulture).
By contrast, [char] instances are compared by their Unicode code points, albeit inconsistently, depending on the specific comparison operator used; note that PowerShell has no [char] literal representation, so a [char] cast is always required:
-eq (aka -ieq) and its case-sensitive variant, -ceq, as well as its negated variants (-ne (aka -ine) and -cne) do take case into account, as implied / requested: If case-insensitivity is required, the operands are first normalized via .ToUpperInvariant() behind the scenes:
# Equivalent to:
# [uint16] [char]::ToUpperInvariant('A') -eq [uint16] [char]::ToUpperInvariant('a')
# Casting to [uint16] returns a [char] instance's Unicode code point.
[char] 'A' -eq [char] 'a' # -> $true
By contrast, -lt (less than) and -gt (greater than) and its variants unexpectedly ignore case-sensitivity:
# !! Unexpectedly ALSO $true
# !! Equivalent to [uint16] [char] 'A' -lt [uint16] [char] 'a'
# !! i.e. NO case normalization is performed.
[char] 'A' -lt [char] 'a' # ditto for -ilt and (as expected) for -clt