apache-sparkkubernetesbigdataspark-submit

How to resolve: `pods is forbidden: User "system:anonymous" cannot watch resource "pods" in API group "" in the namespace "default"`

I want to use Kubernetes as resource manager for spark.

so I wanted to submit a jar far to spark cluster with spark-submit:

./bin/spark-submit  \
    --master k8s://https://vm:6443 \
    --class com.example.WordCounter  \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=default  \
    --conf spark.kubernetes.container.image=private-docker-registery/spark/spark:3.2.1-3 \
    --conf spark.kubernetes.namespace=default \
    --conf spark.kubernetes.authenticate.submission.oauthToken=$TOKEN  \
    --conf spark.kubernetes.authenticate.caCertFile=api.cert \
    java-word-count-1.0-SNAPSHOT.jar

for service account:

kubectl create serviceaccount spark

kubectl create clusterrolebinding spark-role \
    --clusterrole=edit \
    --serviceaccount=default:default \
    --namespace=default

for caCertFile I used the /etc/kubernetes/pki/apiserver.crt content.

and for submission.oauthToken:

kubectl get secret spark-token-86tns -o yaml | grep token

and use the token part.

but still doesn't work and I pods is forbidden: User "system:anonymous" cannot watch resource "pods" in API group "" in the namespace "default" error

Solution

  • spark.kubernetes.authenticate.caCertFile has to be service account cert

    and also spark.kubernetes.authenticate.submission.oauthToken has to be service account token.

    both cert and token can be found in service account secret.