gnu-parallel

How to use GNU Parallel to run a list of command in a file?


I would like to use GNU Parallel to run N commands on P cores. I put each command in a file: ListOfCommand. Each line in that file corresponds to a different command and each line is the complete command that can be run 'as is' at the command-line. There are many more commands than cores. So I would like GNU Parallel to run only P commands at a time, and then whenever a command finishes running, the next command in the file should begin (dynamic scheduling).

It seems like this should be doable with GNU Parallel, but I'm having a hard time digesting the man page. Does anyone know how to do this?

[EDIT] After more digging, this seems to work:

parallel -P 4 '{}' < ListOfCommands


Solution

  • As regards supplying the list of commands, you should be able to use any of the following equivalently:

    parallel -j4 < ListOfCommands
    parallel -j4 -a ListOfCommands
    parallel -j4 :::: ListOfCommands
    

    As regards what is run for each command in the list, you can use {} to identify parameters, but if you specify no command to be run, the line from your file is taken as the command:

    So, if you make your ListOfCommands like this:

    echo A
    echo B
    echo C
    

    You can run the following and the command to run for each job is the echo X line from your commands:

    parallel -j4 :::: ListOfCommands
    A
    B
    C
    

    If you just want to put the parameters for the command to run in a file, you write ListOfParameters like this:

    A
    B
    C 
    

    then put the echo command on the GNU Parallel command line itself as follows and use {}:

    parallel -j4 echo {} :::: ListOfParameters
    A
    B
    C