I've been setting up Kerberos with my Hadoop cluster on Ubuntu 20.04.1 LTS and when I try to reformat the namenode in command line after changing all config files and setting everything up (including principals and keytabs), I'm being met by the error:
Exiting with status 1: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: hdfs/hadoopmaster.406bigdata.com@406BIGDATA.COM from keytab /etc/security/keytabs/hdfs.service.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
This is taking place on my master node, with host name "hadoopmaster". Keytabs are stored in /etc/security/keytabs and when checking the keytabs using klist -t -k -e, the keytab has the correct principal "hdfs/hadoopmaster.406bigdata.com@406BIGDATA"
My hdfs-site.xml file consists of the following properties (includes more, but not included in code below as shouldn't be relevant to the error):
<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/security/keytabs/hdfs.service.keytab</value>
</property
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/hadoopmaster.406bigdata.com@406BIGDATA.COM</value>
</property>
I also have yarn setup with keytabs and principals and that starts up fine (log files have been checked and no errors) and can access the WebUI.
Tried changing filepaths of the keytabs outside of the root directory, double checked /etc/hosts file, the file has correct permissions and ownerships but nothing has helped fix the issue.
What happens when you su hdfs and try and use the keytab? --> does hdfs user have permissions to access the file?