hadoophdfsambarinamenodedatanode

Data-node Alive from ambari isnt stable


we have ambari cluster version - 2.6.1 and hadoop version 2.6.4

number of datanode are - 10

from ambari dashboard we can see the window that show the follwing:

   DataNodes Live

   9/10

but after few min all datanode are alive as:

   DataNodes Live

   10/10

and again after few min we see again

   DataNodes Live

   9/10

its seems that namenode has not received heartbeat msg from a datanode for more than interval, then datanode will be marked and as "dead"

we check the follwing:

  1. host resolution are - OK ( DNS is OK )
  2. IP's resolution are - ok ( DNS is OK )
  3. HDFS service check is passed successfully
  4. each datanode is up ( ps -ef | grep datanode | grep -v grep )
  5. netstat -anp | grep '0.0.0.0:50010' port is ok
  6. systemctl status firewalld.service ( firewall is down as should be )
  7. sestatus SELinux status ( is disable )
  8. MTU is configured to 9000 ( and we verify that 9000 set correctly on all component

what we can do else in order to verify why DataNode alive isn't stable ?


Solution

  • we increased the data node heap size from default value - 2G to 6G since then , now data node alive from ambari dashboard are stable