I have several SBATCH files that use module load to access the required program(s).
Today, after waiting 18 hours in the SLURM queue, a job failed after 3 seconds because it couldn't module load. The cause is that the dependencies of the program I was trying to use were modified by the IT...
I searched through the Environment-Modules documentation for an option to resolve the dependencies automatically but I didn't find anything that worked...
The first workaround that comes to my mind is to request a fixed version of the program, but it's not what I want to do. So I'm thinking of parsing the output of module load to determine the dependencies programmatically.
Here's what I have to type by hand to get, for eg., the "pharokka" program to work:
$ module load pharokka
error: pharokka requires module(s) < PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler > to be loaded first
Loading pharokka/1.7.5
ERROR: Module evaluation aborted
$ module load PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler
error: tRNAscan-SE requires module(s) < infernal > to be loaded first
Loading tRNAscan-SE/2.0.12
ERROR: Module evaluation aborted
$ module load infernal
$ module load PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler
error: minced requires module(s) < graalvm/ce-java11-20.0.0 > to be loaded first
Loading minced/0.4.2
ERROR: Module evaluation aborted
$ module load graalvm/ce-java11-20.0.0
$ module load PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler
$ module load pharokka
How can I automate the whole thing in bash ?
As per @XavierDelaruelle request :
module --version:
Modules Release 4.4.0 (2019-11-17)
pharokka/1.7.5 file content:
#%Module #########
set is_module_rm [module-info mode remove]
#---- whatis is used by module apropos
module-whatis "Set environnement for pharokka (1.7.5)"
module-whatis "topic_0080"
module-whatis "topic_3174"
module-whatis "topic_0160"
module-whatis "topic_0769"
module-whatis "topic_0659"
module-whatis "operation_3672"
module-whatis "operation_3482"
module-whatis "operation_0464"
module-whatis "operation_0335"
module-whatis "operation_0310"
#---- module help message
proc ModulesHelp { } {
puts stderr "This modulefile defines the requisite environement"
puts stderr "needed to use package: pharokka version (1.7.5)"
puts stderr ""
puts stderr "pharokka is designed for rapid standardised annotation of bacteriophages."
puts stderr "URL: https://github.com/gbouras13/pharokka"
puts stderr ""
puts stderr "WARNING: pharokka requires module(s) < PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler >. to be loaded first"
puts stderr ""
## is there any program to document.
puts stderr "package provides following commands:"
puts stderr "\tcreate_custom_hmm.py"
puts stderr "\tinstall_databases.py"
puts stderr "\tpharokka.py"
puts stderr "\tpharokka_multiplotter.py"
puts stderr "\tpharokka_plotter.py"
puts stderr "\tpharokka_proteins.py"
puts stderr ""
}
#---- hack for auto-loading module based on AUTOLOADMODULE env var
if {[info exists env(AUTOLOADMODULE)]} {
module load PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler
}
#---- check for prereq modules
if { [ module-info mode load ] && ! ( [ is-loaded "PHANOTATE/1.5.0" ] && [ is-loaded "MMseqs2/13-45111" ] && [ is-loaded "tRNAscan-SE/2.0.12" ] && [ is-loaded "minced/0.4.2" ] && [ is-loaded "aragorn/1.2.41" ] && [ is-loaded "Mash" ] && [ is-loaded "dnaapler" ] )} {
puts stderr "error: pharokka requires module(s) < PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler > to be loaded first"
exit
}
#---- set PATHs.
prepend-path PATH /opt/.../pharokka/1.7.5/bin
prepend-path PATH /opt/.../pharokka/1.7.5/scripts
Based on the content of the pharokka/1.7.5 modulefile, you should define the AUTOLOADMODULE environment variable to get the dependencies of this module to be declared and loaded automatically
$ export AUTOLOADMODULE=1
However, I need to point that this modulefile should be improve to:
The following section of the modulefile:
#---- hack for auto-loading module based on AUTOLOADMODULE env var
if {[info exists env(AUTOLOADMODULE)]} {
module load PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler
}
#---- check for prereq modules
if { [ module-info mode load ] && ! ( [ is-loaded "PHANOTATE/1.5.0" ] && [ is-loaded "MMseqs2/13-45111" ] && [ is-loaded "tRNAscan-SE/2.0.12" ] && [ is-loaded "minced/0.4.2" ] && [ is-loaded "aragorn/1.2.41" ] && [ is-loaded "Mash" ] && [ is-loaded "dnaapler" ] )} {
puts stderr "error: pharokka requires module(s) < PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler > to be loaded first"
exit
}
Should be rewritten as:
prereq PHANOTATE/1.5.0 MMseqs2/13-45111 tRNAscan-SE/2.0.12 minced/0.4.2 aragorn/1.2.41 Mash dnaapler
This way, the dependencies are correctly expressed and defined in any cases. Then the automated module handling mechanism of Modules is able to automatically loads the module dependencies expressed. It is available since Environment Modules 4.2 and it is enabled by default since version 5.0.
To enable this mechanism:
$ module config auto_handling 1
$ module load pharokka/1.7.5
Or use the --auto command line option:
$ module load --auto pharokka/1.7.5
If this mechanism is disabled, an error will be obtained with a message mentioning the required modules to load before pharokka/1.7.5.