I'm trying to do multiple submissions to condor using one submission script by utilising multiple queue commands. After obviously misunderstanding how this command works with relation to $(Process) I set up the code like so:
executable = exe.sh
arguments = $(Process) arglist1
queue 5
executable = exe.sh
arguments = $(Process) arglist1
queue 17
I had originally thought that after each queue command the $(Process) is reset to 0 and the arguments my script takes relies on this. However after attempting to submit these I found out that this is indeed not the case and that $(Process) just keeps rising. (To make it clear - I would have thought that after queue 5 had finished queuing 5 jobs that when we came to the next instance of $(Process) it would evaluate to 0 and not 5)
My question is, is there a way to reset $(Process) to be 0 after each queue statement? In my actual code I have 28 instances of queue occurring with different numbers of iterations needed in order to pass the right arguments to my executable.
The work around I suppose would be to make 28 separate submission scripts and submit them all with one shell script but i'd rather not go there if possible.
Within one HTCondor cluster of jobs, there is no way to change the Process back to zero, as that is the index of the individual job within the cluster of jobs. Newer versions of HTCondor provide a STEP submit variable for this purpose:
executable = some_executable
arguments = $(STEP) foo bar
queue 5
arguments = $(STEP) foo bar
queue 17
With this syntax, all the jobs will be in one job cluster, which can be condor_rm or condor_hold'ed all at once. The first argument of the first five jobs in the cluster will have argument 0 through 4, then the next 17 will get 0 through 16.