[SOLVED] Unable to submit spark application to k8s spark on window?

I created a Spark cluster on k8s (minikube) and I want to test it by submitting a simple Spark application on Windows.

First I started tunnel for services:

minikube service --all

and get this:

|-----------|------------|-------------|--------------|
| NAMESPACE |    NAME    | TARGET PORT |     URL      |
|-----------|------------|-------------|--------------|
| default   | kubernetes |             | No node port |
|-----------|------------|-------------|--------------|
😿  service default/kubernetes has no node port
|-----------|----------------------|--------------------|---------------------------|
| NAMESPACE |         NAME         |    TARGET PORT     |            URL            |
|-----------|----------------------|--------------------|---------------------------|
| default   | spark-master-service | submit-port/7077   | http://192.168.67.2:30100 |
|           |                      | ui-spark-port/8080 | http://192.168.67.2:30101 |
|-----------|----------------------|--------------------|---------------------------|
|-----------|---------------------|--------------------|---------------------------|
| NAMESPACE |        NAME         |    TARGET PORT     |            URL            |
|-----------|---------------------|--------------------|---------------------------|
| default   | spark-woker-service | ui-spark-port/8081 | http://192.168.67.2:30102 |
|-----------|---------------------|--------------------|---------------------------|
🏃  Starting tunnel for service kubernetes.
🏃  Starting tunnel for service spark-master-service.
🏃  Starting tunnel for service spark-woker-service.
|-----------|----------------------|-------------|------------------------|
| NAMESPACE |         NAME         | TARGET PORT |          URL           |
|-----------|----------------------|-------------|------------------------|
| default   | kubernetes           |             | http://127.0.0.1:61501 |
| default   | spark-master-service |             | http://127.0.0.1:61503 |
|           |                      |             | http://127.0.0.1:61504 |
| default   | spark-woker-service  |             | http://127.0.0.1:61506 |
|-----------|----------------------|-------------|------------------------|
🎉  Opening service default/kubernetes in default browser...
[default spark-master-service  http://127.0.0.1:61503
http://127.0.0.1:61504]
🎉  Opening service default/spark-woker-service in default browser...
❗  Because you are using a Docker driver on windows, the terminal needs to be open to run it.

Then I tried to create a simple spark application on my local machine to submit on this cluster:

from pyspark.sql import SparkSession

def main():
    spark = SparkSession.builder \
        .appName("PrintNumbers") \
        .getOrCreate()

    numbers = list(range(1, 21))
    rdd = spark.sparkContext.parallelize(numbers)

    rdd.foreach(print)

    spark.stop()

if __name__ == "__main__":
    main()

and I ran cmd:

C:\Spark\spark-3.0.0-bin-hadoop3.2>bin\spark-submit --master k8s://http://127.0.0.1:61503 ^
More? --deploy-mode cluster ^
More? --conf spark.executor.instances=1 ^
More? --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark ^
More? --conf spark.kubernetes.container.image=bitnami/spark ^
More? --class org.apache.spark.examples.JavaSparkPi ^
More? --name spark-pi ^
More? "local:///D:/spark-k8s tutorial/spark-submit.py"

and I got error when I was checking work on master:

23/11/19 19:10:35 WARN Utils: Kubernetes master URL uses HTTP instead of HTTPS.
23/11/19 19:10:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/11/19 19:10:35 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
23/11/19 19:10:36 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
23/11/19 19:10:36 WARN WatchConnectionManager: Exec Failure
java.io.IOException: unexpected end of stream on Connection{127.0.0.1:61503, proxy=DIRECT hostAddress=/127.0.0.1:61503 cipherSuite=none protocol=http/1.1}
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:111)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=à
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:236)
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
        ... 28 more
Exception in thread "kubernetes-dispatcher-0" Exception in thread "main" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@69348aff rejected from java.util.concurrent.ScheduledThreadPoolExecutor@369a631[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
        at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.scheduleReconnect(WatchConnectionManager.java:305)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.access$800(WatchConnectionManager.java:50)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:218)
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
        at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
io.fabric8.kubernetes.client.KubernetesClientException: Failed to start websocket
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onFailure(WatchConnectionManager.java:209)
        at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
        at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211)
        at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.Throwable: waiting here
                at io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:144)
                at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:341)
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:755)
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:739)
                at io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:70)
                at org.apache.spark.deploy.k8s.submit.Client.$anonfun$run$1(KubernetesClientApplication.scala:129)
                at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2538)
                at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:129)
                at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4(KubernetesClientApplication.scala:221)
                at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$4$adapted(KubernetesClientApplication.scala:215)
                at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2539)
                at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:215)
                at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:188)
                at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
                at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
                at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
                at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
                at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
                at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
                at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: unexpected end of stream on Connection{127.0.0.1:61503, proxy=DIRECT hostAddress=/127.0.0.1:61503 cipherSuite=none protocol=http/1.1}
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:208)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:111)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
        at okhttp3.RealCall$AsyncCall.execute(RealCall.java:201)
        ... 4 more
Caused by: java.io.EOFException: \n not found: limit=0 content=?
        at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:236)
        at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:215)
        at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
        ... 28 more
23/11/19 19:10:36 INFO ShutdownHookManager: Shutdown hook called
23/11/19 19:10:36 INFO ShutdownHookManager: Deleting directory C:\Users\ABC\AppData\Local\Temp\spark-3e986567-7acd-4561-bf9b-014183f85c64

Spark UI
Can somebody help me please? Thank you very much.

I tried to submit my Spark application and there were something wrong and now I want help for my problem described.