apache-sparkpyspark

How to run a script in PySpark

I'm trying to run a script in the pyspark environment but so far I haven't been able to.

How can I run a script like python script.py but in pyspark?

Solution

  • You can do: ./bin/spark-submit mypythonfile.py

    Running python applications through pyspark is not supported as of Spark 2.0.