bashshellunixslurm

Pass command line arguments via sbatch

Suppose that I have the following simple bash script which I want to submit to a batch server through SLURM:

#!/bin/bash

#SBATCH -o "outFile"$1".txt"
#SBATCH -e "errFile"$1".txt"

hostname

exit 0

In this script, I simply want to write the output of hostname on a textfile whose full name I control via the command-line, like so:

login-2:jobs$ sbatch -D `pwd` exampleJob.sh 1
Submitted batch job 203775

Unfortunately, it seems that my last command-line argument (1) is not parsed through sbatch, since the files created do not have the suffix I'm looking for and the string "$1" is interpreted literally:

login-2:jobs$ ls
errFile$1.txt  exampleJob.sh outFile$1.txt

I've looked around places in SO and elsewhere, but I haven't had any luck. Essentially what I'm looking for is the equivalent of the -v switch of the qsub utility in Torque-enabled clusters.

Edit: As mentioned in the underlying comment thread, I solved my problem the hard way: instead of having one single script that would be submitted several times to the batch server, each with different command line arguments, I created a "master script" that simply echoed and redirected the same content onto different scripts, the content of each being changed by the command line parameter passed. Then I submitted all of those to my batch server through sbatch. However, this does not answer the original question, so I hesitate to add it as an answer to my question or mark this question solved.

Solution

  • The lines starting with #SBATCH are not interpreted by bash but are replaced with code by sbatch. The sbatch options do not support $1 vars (only %j and some others, replacing $1 by %1 will not work). When you don't have different sbatch processes running in parallel, you could try

    #!/bin/bash

touch outFile${1}.txt errFile${1}.txt
rm link_out.sbatch link_err.sbatch 2>/dev/null # remove links from previous runs
ln -s outFile${1}.txt link_out.sbatch
ln -s errFile${1}.txt link_err.sbatch

#SBATCH -o link_out.sbatch
#SBATCH -e link_err.sbatch

hostname
# I do not know about the background processing of sbatch, are the jobs still running
# at this point? When they are, you can not delete the temporary symlinks yet.

exit 0

    Alternative: As you said in a comment yourself, you could make a masterscript. This script can contain lines like

    cat  exampleJob.sh.template | sed -e 's/File.txt/File'$1'.txt/' > exampleJob.sh
# I do not know, is the following needed with sbatch?
chmod +x exampleJob.sh

    In your template the #SBATCH lines look like

    #SBATCH -o "outFile.txt"
#SBATCH -e "errFile.txt"