hadoopspark-streamingkerberoskerberos-delegation

HDFS Delegation token expired even after adding principle to command line

I'm running a spark streaming job in hadoop. The machine is kerberized and it ran fine for 42 Hours and fails due to HDFS token delegation expiry.

I even have the crontab set up for kinit command that runs for every 8 hours and I have the Max-renew lifetime token for 10 hours.

Failed due to error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token

I'm also running the job with principle in the command line while initiating it.

Solution

  • Issue Solved...! Added the following config to spark command line while initiating the job and it worked.

    --conf spark.hadoop.fs.hdfs.impl.disable.cache=true

    or you can change this at yarn config level to impact globally.

    I tested it its running fine for 3 days.

    Thanks