hadoophivenullpointerexceptionmapreduce

Hive Always Fails at Mapreduce


I just installed hadoop 3.3.6 and hive 4.0.0 with mysql as metastore. when running create table or select * from... it runs well. But when I try to do insert or select join, hive always fails. I'm noticing that hive fails at the mapreduce task and "requires lock = true" even though I didn't create any lock.

hiveserver 2 log

I already gave chmod 777 permission to the warehouse directory in hdfs but it didn't solve my problem. yarn doesn't give log when using MR execution Engine

Here's the log that I get from hive.log:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask
No Stats for default@student1, Columns: name, gpa, age
No Stats for default@student2, Columns: nama, jurusan, hobi
Query ID = hadoop_20240923190426_36da4532-a25c-4d21-b7ec-47f18290f3dd
Total jobs = 1
java.lang.NullPointerException  at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:843)
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:458)
    at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:475)
    at org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initialize(MapredLocalTask.java:136)
    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:766)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Execution failed with exit status: 1
Obtaining error information

Task failed!
Task ID:
  Stage-4

Logs:

/tmp/hadoop/hive.log
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

Sorry for my bad English and wrong use of tool in this forum. It's my first time asking here.

update: here's my core-site.xml:

<property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/tmpdata</value>
</property>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
</property>
<property>
    <name>hadoop.proxyuser.hadoop.groups</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.hadoop.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.server.hosts</name>
    <value>*</value>
</property>
<property>
    <name>hadoop.proxyuser.server.groups</name>
    <value>*</value>
</property>

hdfs-site.xml:

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>
<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
<property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>
<property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>

as for my hive-site.xml it's pretty much the same as the template hive gave. just some change for connecting to mysql metastore.

update here's my hive-site.xml:

<configuration>
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://localhost:3306/metastore?createDatabaseIfNotExist=true</value>
    <description>metadata is stored in a MySQL server</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.cj.jdbc.Driver</value>
    <description>MySQL JDBC driver class</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hadoop</value>
    <description>user name for connecting to mysql server </description>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>12345</value>
    <description>password for connecting to mysql server </description>
</property>
<property>
  <name>hive.metastore.uris</name>
  <value>thrift://localhost:9083</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.</description>
</property>
</configuration>

Solution

  • I tried to example job that hadoop give and it didn't work either. turns out my mapred-site.xml configuration was wrong. so I changed the mapred-site.xml to the following configuration:

    <configuration> 
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.map.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    <property>
        <name>mapreduce.reduce.env</name>
        <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
    </property>
    </configuration>
    

    I was pointing HADOOP_MAPREDUCE_HOME direcly into lib folder and that's why every mapreduce job fail