I'm trying to multithread this
$var3 = Get-ItemPropertyValue $var2 -Name fullName,CreationTime,lastAccessTime,LastWriteTime,length;
by replacing it with:
$var3 = forEach($file in $var2) {
start-job -name "bla" -scriptblock {Get-ItemPropertyValue $file.FullName -Name fullName,LastWriteTime,CreationTime,lastAccessTime,length} | out-null
}
where $var2
holds a list of files retrieved via gci
.
This works for normal paths, but not for long UNC Paths prefixed by \\?\UNC\
.
The long path itself works fine with the get-itemPropertyValue '\\?\UNC\some long path in Webdav'
but doesn't as soon as I put it into the scriptBlock from start-job.
The errormessage says (in german):
"Das Argument für den Parameter "Path" kann nicht überprüft werden. Das Argument ist NULL oder leer. Geben Sie ein Argument an, das nicht NULL oder leer ist, und führen Sie den Befehl erneut aus. + CategoryInfo : InvalidData: (:) [Get-ItemPropertyValue], ParameterBindingValidationException + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.GetItemPropertyValueCommand + PSComputerName : localhost
When I try to bind it with get-itemPropertyValue -LocalPath '\\?\UNC\some long path in Webdav'
it also fails but with this Errormessage:
Das Argument kann nicht an den Parameter "LiteralPath" gebunden werden, da es NULL ist. + CategoryInfo : InvalidData: (:) [Get-ItemPropertyValue], ParameterBindingValidationException + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.GetItemPropertyValueCommand + PSComputerName : localhost
Can anyone please check, if it is true, that this doesn't work, or show a running example otherwise? I'm running PS Version 5.1 and have to use this version.
Thanks in advance.
As in my comment, the scope of a job is not the same as your current scope, meaning that, the jobs you start cannot see variables defined outside it's scope (i.e.: $file
). You need to pass the variables to it's scope, either with $using:foo
or with -ArgumentList
:
$props = 'FullName','LastWriteTime','CreationTime','LastAccessTime','Length'
$var3 = forEach($file in (Get-ChildItem -File))
{
Start-Job -Name "bla" -ScriptBlock {
Get-ItemPropertyValue $using:file.FullName -Name $using:props
}
}
$var3 | Receive-Job -Wait -AutoRemoveJob
# OR
$var3 = forEach($file in (Get-ChildItem -File))
{
Start-Job -Name "bla" -ScriptBlock {
param($file, $props)
Get-ItemPropertyValue $file.FullName -Name $props
} -ArgumentList $file, $props
}
$var3 | Receive-Job -Wait -AutoRemoveJob
As for @TheMadTechnician's comment, he is right, there is no need to use Get-ItemProperty
if you're already calling Get-ChildItem
but for the sake of explaining what I meant in my next comment: ...divide the array you're looping through into chunks and passing that chunk to a job instead of file by file as it..., starting a job for each file would be not only many times slower than a normal foreach
loop but also would consume a lot of memory. Start-Job
in general is slower than a linear loop, if you're looking for a multithread alternative, you should be looking at either RunSpace or Start-ThreadJob
from the ThreadJob Module. I have done some testing comparing the performance Linear Loops vs ThreadJob vs Runspace when looping through local directories and files, you can download the script from my GitHub if you're interested (bear in mind, it requires having the ThreadJob module installed).
$directories = Get-ChildItem . -Directory -Recurse
$numberOfThreads = 10
$groupSize = [math]::Ceiling($directories.Count / $numberOfThreads)
$counter = [pscustomobject]@{ Value = 0 }
$groups = $directories | Group-Object -Property {
[math]::Floor($counter.Value++ / $groupSize)
}
$var3 = foreach($chunk in $groups)
{
Start-Job -ScriptBlock {
$folders = $using:chunk.Group.FullName
foreach($folder in $folders)
{
[pscustomobject]@{
DirectoryFullName = $folder
NumberOfFiles = (Get-ChildItem $folder -File).count
}
}
}
}
$result = $var3 | Receive-Job -Wait -AutoRemoveJob |
Sort-Object NumberOfFiles -Descending |
Select-Object * -ExcludeProperty RunspaceID,PSSourceJobInstanceId
If you try this on your computer, you would be getting something like:
DirectoryFullName NumberOfFiles
----------------- -------------
X:\ExampleUser\Folder0 1954
X:\ExampleUser\Folder1 649
X:\ExampleUser\Folder2 64
X:\ExampleUser\Folder3 36
X:\ExampleUser\Folder4 23
X:\ExampleUser\Folder5 16
X:\ExampleUser\Folder6 15
X:\ExampleUser\Folder7 15
X:\ExampleUser\Folder8 12
X:\ExampleUser\Folder9 10