I have installed hadoop 3.1.0 clusters on 4 linux machines, hadoop1(master),hadoop2,hadoop3,and hadoop4.
I ran start-dfs.sh and start-yarn.sh, and saw only namenodes and datanodes running with jps. secondary namenodes, nodemanagers and resourcemanagers failed. I tried a few solutions and this is where I got. How to configure and start secondary namenodes, nodemanagers and resroucemanagers?
About secondary namenodes logs says
java.net.BindException: Port in use: hadoop1:9000
...
Caused by: java.net.BindException: Address already in use
...
About node manager and resource manager log says
2021-02-21 03:29:03,463 WARN org.eclipse.jetty.webapp.WebAppContext: Failed startup of context o.e.j.w.WebAppContext@51d719bc{/,file:///tmp/jetty-0.0.0.0-8042-node-_-any-8548809575065892553.dir/webapp/,UNAVAILABLE}{/node}
com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Error injecting constructor, java.lang.NoClassDefFoundError: javax/activation/DataSource
at org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver.<init>(JAXBContextResolver.java:52)
at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer$NMWebApp.setup(WebServer.java:153)
while locating org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver
I have hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop1:9000</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/app/hadoop/hadoop-3.1.0/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/app/hadoop/hadoop-3.1.0/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hadoop1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hadoop1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hadoop1:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hadoop1:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hadoop1:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1024</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>1</value>
</property>
workers
hadoop1
hadoop2
hadoop3
hadoop4
etc/hosts
192.168.0.111 hadoop1
192.168.0.112 hadoop2
192.168.0.113 hadoop3
192.168.0.114 hadoop4
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
I had jdk15.0.2 installed and it had some sort of problem with hadoop 3.1.0. Later I installed jdk8 and changed java_home. It went all fine!
About secondary node manager, I had hadoop1:9000 for both fs.defaultFS and dfs.namenode.secondary.http-address, and therefore created a conflict. I changed secondary into 9001 and it went all fine!