linuxsshgnu-parallel

How to use different --workdir for different hosts in GNU parallel


I want to execute my script.sh on local machine and on remote machine. However, on local machine the script.sh file is inside ~/dir1/and on remote machine is inside /dir2.

What I tried was: inside ~/dir1/on local machine I run:

  parallel --workdir /dir2 -S remotemachine -S : ./script.sh ::: someargs

On remote machine runs fine, but on local machine parallel also tries to run script.sh inside /dir2


Solution

  • A job does not know on which server it will be run. This is (among other things) due to --retries which will retry the same job, but on a different server.

    So GNU Parallel does not have a way to make the --workdir different depending on server (apart from the relative dirs: . and ...).

    So what can you do instead?

    If one of the dirs only exists on one of the machines, then you can cd to that just before running the script. The cd will fail on the machine which does not have the dir:

    parallel -S server1,: 'cd dir-on-both; cd dir-on-one; script.sh' ::: some args
    

    Use a common symlink:

    ssh server1 ln -s dir1 link1 
    ln -s dir2 link1
    parallel --workdir link1/ -S server1,: script.sh ::: some args
    

    If it is only due to the executable, set PATH to include both dirs:

    PATH=$PATH:dir1:dir2
    parallel --env PATH -S server1,: script.sh ::: some args
    

    Edit 20240422

    You can now use --plus:

    parallel -S localhost,server1,otheruser@server2 --plus echo host:{host} sshlogin:{sshlogin}
    

    These strings will be replaced by $PARALLEL_SSHHOST and $PARALLEL_SSHLOGIN which the shell will evaluate.