powershellparallel.foreach

Identifying batches in PowerShell ForEach -Parallel


I am testing below script to show a separate progress-bar for each batch in ForEach -Parallel execution loop with a limit using -Throttle 5. The intention is to show a separate progress bar foe each batch in the parallel loop. But it doesn't seem to work. How can we identify and assign a unique ID for each batch running in parallel so that I can see a separate progress-bar for each batch?

Workflow WFTest
{
    $ArrOfArr = @(
                    @('a1','a2','a3','a4','a5'),
                    @('b1','b2','b3','b4','b5'),
                    @('c1','c2','c3','c4','c5'),
                    @('d1','d2','d3','d4','d5'),
                    @('e1','e2','e3','e4','e5')
                 )
    
    $UsedID = @()
    $PBID = 1
    ForEach -Parallel -Throttle 5 ($Arr In $ArrOfArr)
    {
        Sequence 
        {
            $CurrentProgress = 0
            $EachItemPCT = 0
            $TotalItems = $Arr.Count

            If($TotalItems -gt 1)
            {
                $EachItemPCT = [Math]::Round(100/$TotalItems, 3)
            }
            Else
            {
                $EachItemPCT = 100
            }

            If($PBID -in $UsedID)
            {
                $WORKFLOW:PBID++
                $WORKFLOW:UsedID += $PBID
                "Using new '$PBID' for batch $($Arr -join ",")"
            }
            else
            {
                "Using ID '$PBID' for batch $($Arr -join ",")"
                $WORKFLOW:UsedID += $PBID
            }


            $ParentActivity = "Processing $TotalItems items in batch $($Arr -join ",")"

            ForEach ($Item in $Arr)
            {
                Write-Progress -Activity $ParentActivity -Status "Processing $Item" -Id $PBID -PercentComplete $CurrentProgress
                Start-Sleep -Seconds 1
                $CurrentProgress += $EachItemPCT
            }
            Write-Progress -Activity "Completed" -Completed -Id $PBID
        }
    }
}

$Start = Get-Date
Write-Output "Started at $($Start.ToString("HH:mm:ss.ffffff"))"
WFTest
$Finish = Get-Date
Write-Output "Finished at $($Finish.ToString("HH:mm:ss.ffffff"))"
Write-Output "Completed in $(([timespan]($Finish-$Start)).TotalSeconds)"

The script gives below output and shows only one progress-bar for all batches running in parallel. All batches are using same ID:

Started at 13:29:40.809269
Using ID '1' for batch e1,e2,e3,e4,e5
Using ID '1' for batch d1,d2,d3,d4,d5
Using ID '1' for batch c1,c2,c3,c4,c5
Using ID '1' for batch b1,b2,b3,b4,b5
Using ID '1' for batch a1,a2,a3,a4,a5
Finished at 13:29:51.380091
Completed in 10.5708217

Solution

  • I have figured it out with some tweaking of the code and it is now able to show separate progress-bars for each batch:

    Here is the updated code:

    Workflow WFTest
    {
        $ArrOfArr = @(
                        @('a1','a2','a3','a4','a5'),
                        @('b1','b2','b3','b4','b5'),
                        @('c1','c2','c3','c4','c5'),
                        @('d1','d2','d3','d4','d5'),
                        @('e1','e2','e3','e4','e5'),
                        @('f1','f2','f3','f4','f5')
                     )
        
        $PBID = 0
        ForEach -Parallel -Throttle 3 ($Arr In $ArrOfArr)
        {
            $WORKFLOW:PBID++
            $NewID = $PBID+1
            "Using PBID: $NewID"
    
            $CurrentProgress = 0
            $EachItemPCT = 0
            $TotalItems = $Arr.Count
    
            If($TotalItems -gt 1)
            {
                $EachItemPCT = [Math]::Round(100/$TotalItems, 3)
            }
            Else
            {
                $EachItemPCT = 100
            }
    
            $ParentActivity = "Processing $TotalItems items in batch $($Arr -join ",")"
    
            ForEach ($Item in $Arr)
            {
                Write-Progress -Activity $ParentActivity -Status "Processing $Item" -Id $NewID -PercentComplete $CurrentProgress
                Start-Sleep -Seconds 1
                $CurrentProgress += $EachItemPCT
            }
            Write-Progress -Activity "Completed" -Completed -Id $NewID
        }
    }
    
    $Start = Get-Date
    Write-Output "Started at $($Start.ToString("HH:mm:ss.ffffff"))"
    WFTest
    $Finish = Get-Date
    Write-Output "Finished at $($Finish.ToString("HH:mm:ss.ffffff"))"
    Write-Output "Completed in $(([timespan]($Finish-$Start)).TotalSeconds) seconds"
    

    Each progress bar is now updated independently with a different ID. Here is the output:

    Started at 14:33:59.274000
    Using PBID: 2
    Using PBID: 3
    Using PBID: 4
    Using PBID: 5
    Using PBID: 6
    Using PBID: 7
    Finished at 14:34:24.787107
    Completed in 25.5131072 seconds