apache-sparkmesos

spark-submit --status with mesos master returns nothing

I'd like to retrieve the status of a spark job running in cluster mode on a mesos master via the following:

spark-submit --master mesos://<ip>:7077 --status "driver-...-..."

It exits 0 with no logging, no matter what the driver's status is.

I know that it's doing something right, since if I run the command with a an invalid mesos ip/port, I get

Exception in thread "main" org.apache.spark.deploy.rest.SubmitRestConnectionException: Unable to connect to server
at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$requestSubmissionStatus$3.apply(RestSubmissionClient.scala:165)

and if I run with an invalid submission id, I get

2018-10-02 18:47:01 ERROR RestSubmissionClient:70 - Error: Server responded with message of unexpected type SubmissionStatusResponse.

Any idea why spark-submit --status isn't returning anything?

Solution

  • I found a workaround by accessing the dispatcher's api directly:

    curl -s "http://$DISPATCHER/v1/submissions/status/$SUBMISSION_ID"

    Still no clear answer why spark-submit --status does not behave as documented though.