pythonnextflownextflow-tower

Simple Python Script in Nextflow fails


I am attempting to run Python from inside a Nextflow pipeline launched via Nextflow-Tower. I forked the nextflow-io/hello repo and made sure I could launch it successfully from my own repo. What I am doing wrong? If there is a simple python pipeline repo that can be launched(foolproof) using Nextflow-Tower that would be helpful.

My dir structure is:

root
--main.nf
--nextflow.config
--bin
----test.py

nextflow.config

process.container = "us-east1-docker.pkg.dev/xxxx-yyyy/pass/python3:latest"
docker.enabled = true

main.nf

#!/usr/bin/env nextflow
nextflow.enable.dsl=2 

process sayHello {
  input: 
    val x
  output:
    stdout
  script:
    """
    echo '$x world!'
    """
}

process processPython {
  input: 
    val y
  output:
    stdout
  script:
    """
    #!/usr/bin/env python3
    test.py
    """
}

workflow {
  Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') | sayHello | view

  Channel.of('foo', 'bar', 'cluster', 'frack') | processPython | view
}

test.py

import sys
print(f"Python Version: {sys.version}")

Error report

Error executing process > 'processPython (1)'

Caused by:
  Process `processPython (1)` terminated with an error exit status (127)

Command executed:

  #!/usr/bin/env python3
  test.py

Command exit status:
  127

Command output:
  (empty)

Command error:
  /usr/bin/env: 'python3': No such file or directory

Work dir:
  gs://nf-tower-6897ad32-d3d4-4bda-ade1-97d25c0e680c/scratch/4wCvYTX4GV6S3k/98/707cf12d829d888f4f5179ee1ae55e

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Solution

  • The solution was to include shebang IN test.py:

    #!/usr/bin/env python3
    import sys
    print(f"Python Version: {sys.version}")
    

    process:

    process processPython {
    
      container = 'us-east1-docker.pkg.dev/xxxx-yyyy/pass/python3:latest'
    
      input: 
        val y
      output:
        stdout
      script:
        """
        python3 --version
        test.py
        """
    }