Dears,
Apache Hive 3.1.2 with Hadoop 3.1.1 was working fine with until i configured Hive with Tez based on this doc :[ https://github.com/NitinKumar94/Installing-Apache-Tez]
it always gives this error, i tried many solutions with no luck
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/BatchListingOperations
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3268)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3313)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
at org.apache.hadoop.hive.common.FileUtils.getJarFilesByPath(FileUtils.java:1006)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5198)
at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Any help. i tried the following also with no luck
Copy hive-exec jar from $HIVE_HOME/lib to hdfs dir /user/tez/
If any of below errors and yours you can try this solution
Error: Could not find or load main class
org.apache.tez.dag.app.DAGAppMaster
You need to export your tez confs in hive server instance ( hive env sh)
export TEZ_CONF_DIR=/etc/tez/conf/
export TEZ_JARS=/your_path_2_tez/tez-0.10.2/
export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH}
Then you need to copy your tez folder to hdfs but not zip , you need to copy uncompressed tez package
hdfs dfs -mkdir /tez/tez-0.10.2
Then copy your files
hdfs dfs -copyFromLocal /your_path_2_tez/tez-0.10.2/* /tez/tez-0.10.2/
Then define this path in tez-site xml
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/tez/tez-0.10.2,${fs.defaultFS}/tez/tez-0.10.2/lib</value>
</property>
Also you need to copy your hadoop version jars (yours 3.1.1) to tez lib folder :
hadoop-common-3.3.1.jar
hadoop-yarn-common-3.3.1.jar
hadoop-yarn-client-3.3.1.jar
...... .....
the main idea is which class is not exist looking your exception you can copy the jar from your hadoop share jars from your hadoop installation directory. You need to copy this jars to hdfs as well
Then you can restart your hive server
Here you can use tez without error