[SOLVED] Snakemake access snakemake.config in profile config.yaml file

Snakemake access snakemake.config in profile config.yaml file

I want to run a pipeline on a cluster where the name of the jobs are of the form : smk-{config["simulation"]}-{rule}-{wildcards}. Can I just do :

snakemake --profile slurm --configfile myconfig.yaml

With slurm/config.yaml being :

executor: slurm
jobs: 64
latency-wait: 30
cluster:
    sbatch
        --partition=workq
        --job-name=smk-{config["simulation"]}-{rule}-{wildcards}
        [...]

Solution

Assuming you are using the latest Snakemake, you don't want to mix the executor: slurm and the cluster: sbatch ... options. You only need executor: slurm. The cluster options relate to what is now the cluster-generic executor plugin, but you only need to use the slurm executor plugin.

Unfortunately, while this is the recommended executor, this does not help with you setting the job name because the slurm executor plugin sets the job name to a UUID and it cannot be changed.

I'd advise you to try out a third option which is the DRMAA plugin. It works really well for me (I'm maintaining pipelines for a genome sequencing core facility so I run a lot of Snakemake jobs on SLURM).

Here's a config snippet:

default-resources:
- time_h=24
- mem_mb=6000
- n_cpus=1
- slurm_partition='standard'
- slurm_qos='your_qos'
- slurm_account='your_account'
- slurm_extra=''
executor: drmaa
drmaa-args: -p standard --qos={resources.slurm_qos} --account={resources.slurm_account}
  --time={resources.time_h}:00:00 --mem={resources.mem_mb} --mincpus={resources.n_cpus}
  {resources.slurm_extra}
jobname: 'smk-{config[simulation]}-{rule}-{wildcards}-{jobid}'

I've tested this on the latest Snakemake, and job names are being set as expected. You will need to have the slurm-drmaa driver installed on your headnode - see https://github.com/natefoo/slurm-drmaa