gnu-parallel

How to use GNU Parallel to run a list of command in a file?

I would like to use GNU Parallel to run N commands on P cores. I put each command in a file: ListOfCommand. Each line in that file corresponds to a different command and each line is the complete command that can be run 'as is' at the command-line. There are many more commands than cores. So I would like GNU Parallel to run only P commands at a time, and then whenever a command finishes running, the next command in the file should begin (dynamic scheduling).

It seems like this should be doable with GNU Parallel, but I'm having a hard time digesting the man page. Does anyone know how to do this?

[EDIT] After more digging, this seems to work:

parallel -P 4 '{}' < ListOfCommands

Solution

  • As regards supplying the list of commands, you should be able to use any of the following equivalently:

    parallel -j4 < ListOfCommands
parallel -j4 -a ListOfCommands
parallel -j4 :::: ListOfCommands

    As regards what is run for each command in the list, you can use {} to identify parameters, but if you specify no command to be run, the line from your file is taken as the command:

    So, if you make your ListOfCommands like this:

    echo A
echo B
echo C

    You can run the following and the command to run for each job is the echo X line from your commands:

    parallel -j4 :::: ListOfCommands
A
B
C

    If you just want to put the parameters for the command to run in a file, you write ListOfParameters like this:

    A
B
C

    then put the echo command on the GNU Parallel command line itself as follows and use {}:

    parallel -j4 echo {} :::: ListOfParameters
A
B
C