for-loopcmdcommand-linewaitmultitasking

From CMD, is it possible to execute a command after a for loop using start to launch several parallel tasks, but only AFTER all tasks have completed?


I'm running a command that modifies all the audio files in a folder, due to a bug in ffmpeg, I need to chain together an ffmpeg and sox command, but sox is really slow, so I run them in parallel, which dramatically speeds up the process (makes it about 10 times faster, probably the more cores in your CPU, the faster it will go). That all works fine.

The problem is that because those are all spawned as separate tasks, it returns and runs the commands after the FOR loop before it has completed the tasks. In my case, that's to delete the intermediate temp files (del int_*.wav in the example below). Here's a simplified example (removed most parameters to make it easier to read):

md "Ready" & (for %x in ("*.mp3") do (start "Convert" cmd /c "ffmpeg -i "%x" -f wav "int_%x.wav" & sox "int_%x.wav" -r 44100 "Ready\ready-%x"")) && del int_*.wav

That converts all the MP3 files in the current directory per my parameters to a different set of MP3 files in the destination directory (the Ready subdirectory). It uses .WAV files as intermediaries because those are lossless and fast.

The above runs correctly, except for the last part, del int_*.wav, because it tries to run that before all the START threads have finished, and so it does nothing.

It errors with:

Could Not Find C:\<parent directory name>\int_*.wav

Which is expected, because it gets to the DEL before it has created the files to delete. Hence my need to have it pause until the FOR loop has completed and all the START tasks have closed.

It does run the DEL command correctly if I leave out the START command and don't try to run them in parallel, but then it's very slow (this is how I originally did it, then came up with the above solution to make it faster):

md "Ready" & (for %x in ("*.mp3") do (ffmpeg -i "%x" -f wav "int_%x.wav" & start sox "int_%x.wav" -r 44100 "Ready\ready-%x")) & del int_*.wav

And note that this example only has the DEL command at the end, but in my actual use case, I have multiple commands after the loop, including a batch file, all chained together with & signs. So I can't use a solution that just provides a different way to delete the files. I need to have everything after the double closing parentheses wait to run until the loop has completed and all the started tasks have closed.

Also note that I'm intentionally running this as one command from the command prompt rather than via a batch file. This is because I have to change a small varying subset of many parameters and generally run it on a subset of the files in a directory. So in this particular case, it's much easier to just paste the command and tweak the relevant parameters and filenames with wildcards than make a batch file and have to pass it everything every time. I'd like to keep it as a single CMD line if possible.

Is there any way to do that?


Solution

  • Avoid cmd these days. Everything is much easier in PowerShell, and many things that are impossible to do in cmd can be achievable in PowerShell

    foreach (f in Get-ChildItem -Filter "*.mp3") {
        Start-Job -ScriptBlock {
            ffmpeg -i "$f" -f wav "int_$f.wav"
            sox "int_$f.wav" -r 44100 "Ready\ready-$f"
        }
    }
    Get-Job | Wait-Job
    Remove-Item int_*.wav
    

    Or you can pipe directly without Get-Job like this

    foreach (f in ls "*.mp3") { sajb {
        ffmpeg -i "$f" -f wav "int_$f.wav"; sox "int_$f.wav" -r 44100 "Ready\ready-$f"
    } } | wjb
    del int_*.wav
    

    Or even make it a one-liner

    foreach (f in ls "*.mp3") { sajb { ffmpeg -i "$f" -f wav "int_$f.wav"; sox "int_$f.wav" -r 44100 "Ready\ready-$f" } } | wjb; rm int_*.wav
    

    So in this particular case, it's much easier to just paste the command and tweak the relevant parameters and filenames with wildcards than make a batch file and have to pass it everything every time

    Nothing prevents you from setting a default value if the parameter is not set. This is possible in any shell scripting languages. Just pass the arguments that you need to change. However in batch you'll need to read arguments and set the flags manually while in PowerShell just define what you want with Param() and everything will be handled automatically


    Alternatively you can use Start-Process/Wait-Process instead of Start-Job/Wait-Job but this is longer, less readable and may not work for some commands

    Start-Process -PassThru {
        cmd /c "ffmpeg -i `"$f`" -f wav `"int_$f.wav`" & sox `"int_$f.wav`" -r 44100 `"Ready\ready-$f`""
    } | Wait-Process
    rm int_*.wav