I'm using Datastax Enterprise 4.8.3. I'm trying to implement a Quartz based application to remotely submit Spark jobs. During my research I have stumbled upon the following links:
To test out the theory I tried executing the below code snippet on the master node (IP: "spark-master-ip"; directly on the shell) of my 2 node cluster (as provided in link #1 above):
curl -X POST http://spark-master-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "myAppArgument1" ],
"appResource" : "file:/home/local/sparkjob.jar",
"clientSparkVersion" : "1.4.2",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "com.spark.job.Launcher",
"sparkProperties" : {
"spark.jars" : "file:/home/local/sparkjob.jar",
"spark.driver.supervise" : "false",
"spark.app.name" : "MyJob",
"spark.eventLog.enabled": "true",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://spark-master-ip:6066"
}
}'
But executing the code I get an html response with the following text:
This Page Cannot Be Displayed
The system cannot communicate with the external server (spark-master-ip).
The Internet server may be busy, may be permanently down, or may be unreachable because of network problems.
Please check the spelling of the Internet address entered.
If it is correct, try this request later.
If you have questions, please contact your organization's network administrator and provide the codes shown below.
Date: Fri, 11 Dec 2015 13:19:15 GMT
Username:
Source IP: spark-master-ip
URL: POST http://spark-master-ip/v1/submissions/create
Category: Uncategorized URLs
Reason: UNKNOWN
Notification: GATEWAY_TIMEOUT
Check that you have started a Spark master and worker (using start-all.sh
)
Check that in the log file there is a message like
INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
netstat
)It should look like this:
root@x:~# netstat -apn | grep 11572 | grep LISTEN tcp6 0 0 :::8080 :::* LISTEN 11572/java tcp6 0 0 10.0.0.9:6066 :::* LISTEN 11572/java tcp6 0 0 10.0.0.9:7077 :::* LISTEN 11572/java
Then replace "spark-master-ip" in the script with the IP address you see in the output of netstat
(the example shows "10.0.0.9").