powershellparallel-processingfile-copying

Copying multiple files in parallel through power-shell without using any third party software?


Problem Statement: I am trying to copy 100 of files (each of them like more than a GB in size) from source to the destination directory, I am automating this by a power-shell script. While executing the script the copy operation is copying the files in sequence. Is there any way we can copy them in parallel to reduce some time as it is taking a lot of time to copy all the files & have a limitation of using any third-party software.

    $DATAFileDir="D:\TEST_FOLDER\DATAFILESFX\*"
    $LOGFileDir="D:\TEST_FOLDER\LOGFILESFX\*"
    $DestDataDir="D:\TEST_FOLDER\Data\"
    $DestLogDir="D:\TEST_FOLDER\Log\"

    #Copying the Primary file
    Copy-Item -Path $DATAFileDir -Destination $DestDataDir -Recurse -Force -Verbose
    #Copying the Audit File
    Copy-Item -Path $LOGFileDir -Destination $DestLogDir -Recurse -Force -Verbose

Any suggestion for it ?


Solution

  • You can start job individual process for every file you want to copy.

    $Source = Get-ChildItem -Path C:\SourceFolder -Recurse | Select -ExpandProperty FullName
    $Destination = 'C:\DestinationFolder'
    foreach ($Item in @($Source)){
        #starting job for every item in source list
        Start-Job -ScriptBlock {
            param($Item,$Destination) #passing parameters for copy-item 
                #doing copy-item
                Copy-Item -Path $Item -Destination $Destination -Recurse  -Force
        } -ArgumentList $Item,$Destination #passing parameters for copy-item 
    }