powershellcsvout-of-memory

Memory exception while filtering large CSV files


enter image description heregetting memory exception while running this code. Is there a way to filter one file at a time and write output and append after processing each file. Seems the below code loads everything to memory.

$inputFolder = "C:\Change\2019\October"
$outputFile = "C:\Change\2019\output.csv"
Get-ChildItem $inputFolder -File -Filter '*.csv' |
    ForEach-Object { Import-Csv $_.FullName } |
    Where-Object { $_.machine_type -eq 'workstations' } |
    Export-Csv $outputFile -NoType


Solution

  • May be can you export and filter your files one by one and append result into your output file like this :

    $inputFolder = "C:\Change\2019\October"
    $outputFile = "C:\Change\2019\output.csv"
    
    Remove-Item $outputFile -Force -ErrorAction SilentlyContinue
    
    Get-ChildItem $inputFolder -Filter "*.csv" -file | %{import-csv $_.FullName | where machine_type -eq 'workstations' | export-csv $outputFile -Append -notype }