[SOLVED] Spark Mesos cluster setting a wrong route for spark-class in executors

Spark Mesos cluster setting a wrong route for spark-class in executors

I have a flask API that using pyspark starts spark and sends the job to a mesos cluster.

The executor fails because it's taking part of the route where spark-class is located in the Flask API.

Logs:

I0903 13:16:14.295672  4754 exec.cpp:164] Version: 1.9.0
I0903 13:16:14.299304  4760 exec.cpp:237] Executor registered on agent 3ac2aa4b-ff5b-4340-a56e-efe96b3b07a2-S2
I0903 13:16:14.301093  4758 executor.cpp:190] Received SUBSCRIBED event
I0903 13:16:14.301807  4758 executor.cpp:194] Subscribed executor on XXXXX
I0903 13:16:14.301923  4758 executor.cpp:190] Received LAUNCH event
I0903 13:16:14.302549  4758 executor.cpp:722] Starting task 3
I0903 13:16:14.310398  4758 executor.cpp:738] Forked command at 4764
sh: /home/user/source/environment/lib/python3.7/site-packages/pyspark/./bin/spark-class: No such file or directory
I0903 13:16:14.397004  4760 executor.cpp:1039] Command exited with status 127 (pid: 4764)
I0903 13:16:15.398034  4763 process.cpp:935] Stopped the socket accept loop

The bold path is the part where the spark-class actually is in the Flask server (requester of the job)

sh: /home/user/source/environment/lib/python3.7/site-packages/pyspark/./bin/spark-class: No such file or directory

Here:

-rwxr-xr-x 1 root root 3.2K Sep  1 11:07 /home/user/source/environment/lib/python3.7/site-packages/pyspark/bin/spark-class

Maybe I am missing an environment variable or can I send the path as a parameter?

Solution

Solved by adding the path where the spark is located in executor by adding this property pointing at the binaries of Spark:

spark.mesos.executor.home