hadoopapache-sparkkerberosgssapikeytab

Spark 1.6.Token can be issued only with kerberos or web authentication

I call kinit keytab right before spark-submit in my shell driver script. The thing is, its working by itself, but when I call the shell driver scrip it through Oozie, I got this error:

Stdoutput py4j.protocol.Py4JJavaError: An error occurred while calling 
o49.saveAsTextFile.
Stdoutput : org.apache.hadoop.ipc.RemoteException(java.io.IOException): 
Delegation Token can be issued only with kerberos or web authentication

The issue is probably here

file.coalesce(1,True).saveAsTextFile(FQDNofHadoop+output)

EDIT: In my script is: kinit -k -t /home/me/me.keytab me@DOMAIN.HAD

EDIT: Working solution:

I used

spark-submit --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \

and execution of pyspark script in oozie worked with no error (even writing into hive table and so on). Log4j logger didn't work (with standalone scripts it does), but at least print() (stdout into yarn logs) yes ...

Thx

Solution

  • I used --principal 'me@DOMAIN.HAD' \ --keytab '/home/me/me.keytab' \ and execution of pyspark script worked with no error (even writing into hive table and so on). Logger didn't work, but at least print() yes ...