I just installed hadoop 3.3.6 and hive 4.0.0 with mysql as metastore. when running create table or select * from... it runs well. But when I try to do insert or select join, hive always fails. I'm noticing that hive fails at the mapreduce task and "requires lock = true" even though I didn't create any lock.
I already gave chmod 777
permission to the warehouse directory in hdfs but it didn't solve my problem.
yarn doesn't give log when using MR execution Engine
Here's the log that I get from hive.log
:
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
No Stats for default@student1, Columns: name, gpa, age
No Stats for default@student2, Columns: nama, jurusan, hobi
Query ID = hadoop_20240923190426_36da4532-a25c-4d21-b7ec-47f18290f3dd
Total jobs = 1
java.lang.NullPointerException at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:843)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:458)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:475)
at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initialize(MapredLocalTask.java:136)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:766)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Execution failed with exit status: 1
Obtaining error information
Task failed!
Task ID:
Stage-4
Logs:
/tmp/hadoop/hive.log
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
Sorry for my bad English and wrong use of tool in this forum. It's my first time asking here.
update: here's my core-site.xml:
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmpdata</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.server.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.server.groups</name>
<value>*</value>
</property>
hdfs-site.xml:
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
as for my hive-site.xml it's pretty much the same as the template hive gave. just some change for connecting to mysql metastore.
update here's my hive-site.xml:
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
<description>metadata is stored in a MySQL server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.cj.jdbc.Driver</value>
<description>MySQL JDBC driver class</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hadoop</value>
<description>user name for connecting to mysql server </description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>12345</value>
<description>password for connecting to mysql server </description>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://localhost:9083</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
</configuration>
I tried to example job that hadoop give and it didn't work either. turns out my mapred-site.xml configuration was wrong. so I changed the mapred-site.xml to the following configuration:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
</configuration>
I was pointing HADOOP_MAPREDUCE_HOME direcly into lib folder and that's why every mapreduce job fail