makefilelsf

LSF Job Array in Make File

I am running an LSF job array to create a target in a makefile. However as soon as the array is submitted make considers the command for the target to be executed, and throws an error as the target does not exist.

How can I force make to wait until the completion of the LSF job array before moving onto other dependent targets?

Example:

all: final.txt

first_%.txt:
    bsub -J" "jarray[1-100]" < script.sh

final.txt: first_%.txt
    cat first_1.txt first_50.txt first_100.txt > final.txt

Unfortunately the -K flag isn't supported for job arrays.

Solution

  • Try bsub -K which should force bsub to stay in the foreground until the job completes.

    Edit

    Since the option isn't supported on arrays, I think you'll have to submit your array as separate jobs, something like:

    for i in `seq 1 100`; do 
            export INDEX=$i
            bsub -K < script.sh & 
done
wait

    You'll have to pass the index to your script manually instead of using the job array index.