Nextflow: mkdir: cannot create directory 'filename': File exists

I'm trying to setup a workflow using Nextflow on WSL2. After some changes, here is the full updated code:

params.output_dir = "./workflow_output"
params.input_dir = null

process fastQC {
    publishDir(
        path: "$params.output_dir/qc_reports",
        mode: "copy"
    )

    input:
    val fqs

    output:
    path "$params.input_dir/*_fastqc.zip", emit: zip
    path "$params.input_dir/*_fastqc.html", emit: html

    """
    fastqc $fqs
    """
}

workflow onlyQC {
    out_dir = file("$params.output_dir")
    fqs = channel.from(file("$params.input_dir/*.fastq"))
    
    fastQC(fqs) 
}

Before changing the fastqc process to use publishDir, I was receiving the following error message when I attempted to use mkdir -p $out_dir prior to running the fastqc -o $out_dir $fqs command.

mkdir: cannot create directory ‘workflow_output’: File exists

I changed to the updated code by using the publishDir directive, but now I get the error:

Missing output file(s) data/*_fastqc.zip expected by process onlyQC:fastQC (3)

Any guidance/help would be appreciated. Thanks!

mkdir: cannot create directory ‘workflow_output’: File exists

You get this error because you are asking Nextflow to stage the output directory (i.e. out_dir) into the process working directory (under ./work), and, by default, input files and directories will be symbolic linked (this can be changed using the stageInMode directive). Note that mkdir will not follow the symlink but instead interpret it as a file, hence the error.

A better way to publish process outputs is to use the publishDir directive. For example:

params.fastqs = "/path/to/*.fastq"
params.output_dir = "./workflow_output"


process fastQC {

    publishDir(
        path: "${params.output_dir}/fastQC",
        mode: 'copy',
    )

    input:
    path fqs

    output:
    path "*_fastqc.zip", emit: zip
    path "*_fastqc.html", emit: html

    """
    fastqc ${fqs}
    """
}

workflow {

    fastqs = Channel.fromPath("${params.fastqs}")

    fastQC( fastqs )
}