pythonapache-sparkpbs

EnvironmentError: Not in a PBS job

I'm trying to queue some scripts on a remote cluster node (rh7) with this code:

#!/bin/bash
#PBS -N spark-cluster-path
#PBS -l select=2:ncpus=4:mem=2G
#PBS -l walltime=01:00:00

# Qsub template
# Scheduler: PBS

#Environment
export JAVA_HOME=/some/path/to/jdk/1.8.0_112
export SPARK_HOME=/some/path/to/spark/2.2.1
export PYTHONPATH=/some/path/to/python3.6/dist-packages
export PYSPARK_PYTHON=/some/path/to/Python/3.6.5/bin/python3
export PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON
export PATH=$JAVA_HOME/bin:$SPARK_HOME/bin:$PYTHONPATH:$PATH

/some/path/to/spark/spark-on-pbs-0.1/pbs-spark-launcher ./word_count.py

The variables are all correct, the script word_count.py works perfectly fine independantly, but when I run this script I get the following error:

File "/some/path/to/spark/spark-on-pbs-0.1/pbs-spark-launcher", line 118, in <module>
    raise EnvironmentError("Not in a PBS job")
EnvironmentError: Not in a PBS job

Google is drawing a blank here and so am I. Where could this issue come from?

Solution

  • Quoting https://www.osc.edu/~troy/pbstools/man/pbs-spark-submit :

    ASSUMPTIONS AND LIMITATIONS

    pbs-spark-submit makes two assumptions about its environment. First, the Spark master process will be run on the PBS job's mother superior node. Second, the working directory for the Spark programs is on a file system shared across on nodes allocated to the job.

    and by looking at the code it looks like is expecting an environment variable PBS_JOBID