arrayspowershellforeach

Foreach loop that creates ps-objects and write them into new separates arrays gets progressively slower


In my script I try to separate a csv that I imported into an array, into 11 different arrays. I do this by checking a id in each array object which decides into which new array the data should be transferred. I achieve this by creating new ps-objects with the relevant information and then adding them to the other arrays.

The source file I have is 700.000 lines long and the problem is that my foreach loop in which I separate the data gets progressively slower. I tested with 5000 lines and they were through in a few seconds, but with the complete data set I had to cancel the execution of the script after 1:15h.

This is my script:

$Nest1 = @()
$Nest2 = @()
$Nest3 = @()
$Nest4 = @()
$Nest5 = @()
$Nest6 = @()
$Nest7 = @()
$Nest8 = @()
$Nest9 = @()
$Nest10 = @()
$Errornest = @()
$counter = 0

$Pfad = 'C:\Users\Measurements\Documents\Data.csv'
$Header = 'Type','Date','Time','Nestid','Value','goempty'

$Data = Import-Csv $pfad -Delimiter ";" -Header $Header

$Data = $Data | where-Object {$_.Value -ne "Waage antwortet nicht" -and $_.Date -ne ""} 

$total = $Data.count

Foreach ($Dat in $Data){
$counter ++
Write-Progress -Activity 'Processing Data' -PercentComplete (($counter / $total) * 100)

    if($Dat.Nestid -eq 1){
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest1 += $Nestobject
    }
    elseif($Dat.Nestid -eq 2){
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest2 += $Nestobject
    }
    elseif($Dat.Nestid -eq 3){       
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest3 += $Nestobject
    }
    elseif($Dat.Nestid -eq 4){
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest4 += $Nestobject
    }
    elseif($Dat.Nestid -eq 5){
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest5 += $Nestobject}
    elseif($Dat.Nestid -eq 6){
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest6 += $Nestobject}
    elseif($Dat.Nestid -eq 7){
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest7 += $Nestobject}
    elseif($Dat.Nestid -eq 8){      
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest8 += $Nestobject}
    elseif($Dat.Nestid -eq 9){       
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest9 += $Nestobject}
    elseif($Dat.Nestid -eq 10){       
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Nest10 += $Nestobject}
    else{       
       $Nestobject = New-Object psobject
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Type' -Value $Dat.type
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Date' -Value $Dat.Date
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Time' -Value $Dat.Time
       $Nestobject | Add-Member -MemberType NoteProperty -name 'Value' -Value $Dat.Value
       $Errornest += $Nestobject}
}

I already thought about writing a function for creating the accordig object, but i guess that would only bring better readability to the script and not really increase the performance.

Just in case the Data structure i have also is relevant, the source csv looks something like this:

I;17.07.2023;11:16:32;9;000734d9c1;
W;17.07.2023;11:16:31;7;120.79;g                   
W;17.07.2023;11:16:31;3;Waage antwortet nicht;
W;17.07.2023;11:16:32;4;5.08;g                   
W;17.07.2023;11:16:33;10;148.49;g                   
W;17.07.2023;11:16:32;2;49.91;g                   
W;17.07.2023;11:16:33;9;117.13;g                   
I;17.07.2023;11:16:33;9;000734d9c1;
I;17.07.2023;11:16:34;9;000734d9c1;

Solution

  • For starters, it's a very bad practice to paste examples as screenshots. Why don't you use a design like:

    $Nest1 = $Data | Where-Object {$_.Nestid -eq '1'} | Select Type, Date, Time, Value
    

    This should work faster. But I think you first need to understand why 11 separate arrays are needed in the first place.