cluster-computingslurmsnakemake

How to run Snakemake >= 8 on a Slurm cluster?


When I run a snakemake workflow locally with snakemake --cores 1, everything works fine. However I need to use SLURM. I tried the--cluster command that I've seen in the documentation and in nearly every tutorial:

snakemake: error: unrecognized arguments: --cluster

I also tried --slurm, not recognized either. --cluster-config also doesn't appear to be understood by Snakemake.

Looking for some explanation I found that apparently Snakemake 8 made breaking changes. It looks like now things are like this:

New designed: Now if you want to use --cluster CMD, please use --executor cluster-generic --cluster-generic-submit-cmd CMD instead.

Note you should install cluster-generic using command pip install snakemake-executor-cluster-generic

Neither pip nor conda knows a snakemake-executor-cluster-generic package. How do I work with slurm, then?


Solution

  • I ran into a very similar issue. I used to do something like this for a SGE/UGE cluster:

    snakemake \
            --cluster-config cluster.json \
            -j 1 \
            --cluster 'qsub -q all.q -l h_vmem={cluster.mem} -pe smp {cluster.num_cores} -V -cwd'
    

    Where the contents of cluster.json were:

    {
      "__default__": {
        "mem": "8G",
        "num_cores": "{threads}",
        "jobname": "{rule}"
      }
    }
    

    In Snakemake 8, it appears you first need to install a new plugin by going:

    pip install snakemake-executor-plugin-cluster-generic
    

    The --cluster-config argument is no longer available. So the first command becomes:

     snakemake \
             --executor cluster-generic \
             --cluster-generic-submit-cmd 'qsub -N {rule} -q all.q -l h_vmem=8G -pe smp {threads} -V -cwd'
    

    This logic should work for other clusters (e.g. SLURM).