powershellget-childitemfileinfo

Saving file list to file slow


This post is provided to hopefully save anyone saving FileInfo object to file without taking excessive time.

I am building tool to sync files between USB and network storage using robocpy. Normally files that don't exist on target would be copied and that don't exist on source would be deleted (with /PURGE or /MIR). I want to be able to do two way sync, so need to track files that previously were in the set and if missing from either side remove them from the other. This emulates software such as FreeFileSync, but can only be done in Powershell due OS restrictions for us.

To track the files, I am dumping FileInfo object from Get-ChildItem <Path> -Recurse -File to CSV file. The FileInfo object contains 12,000+ objects, and was taking over 3 mins to save to file. Note the time was not due to file write, even using ConvertTo-XML memory object took over 3 mins.

Searching returned a lot of info about Get-ChildItem being slow and recommending .NET EnumerateFiles, but nothing about converting to XML/JSON being slow. My GCI completes in under 5 seconds which is fine for our requirements, it is the conversion (specifically VersionInfo) that takes the extra 3 mins. I will look at EnumerateFiles, as do like to squeeze the best performance out of my scripts. :)


Solution

  • Using select FullName,Length,LastWriteTime reduced the save time to under 3 seconds. Further investigation identified it was the ScriptProperty VersionInfo that was causing the slowdown. Using select with all other FileInfo properties resulted in a save time still under 3 seconds.

    Hope this helps someone.