gnu-parallel

Write output to stdout as well as results in GNU parallel

I'm using GNU parallel to run multiple python scripts in parallel and each one can take a few minutes to complete. The scripts print out their progress to stdout.

An example script.py

print('doing something...')
do_something()
print('doing another thing...')
do_another_thing()

My parallel command which only saves output to a results directory tree

parallel --lb --results results < parallel-commands.txt

The content of parallel-commands.txt

python3 script.py --once
python3 script.py --twice

I want to print the output when it's ready, and I looked online and found the --lb option for parallel, which helped me, but I also want parallel to write the output to a file as well as to stdout, kind of like what tee does. Is there any way to achieve that?

Solution

  • Currently GNU Parallel only supports a single output stream. In other words you cannot get output in both stdout and files.

    So you need to do it manually like this:

    #!/bin/bash

doit() {
    echo $1 start
    sleep $1
    echo $1 end
}
export -f doit

parallel --lb 'doit {} | tee {#}.out' ::: 3 2 1

    It is, however, a feature that is under consideration, so it is good to know that there will be atleast 1 user using it :)