hiveapache-tezhadoop3

Hive with TEZ failed to start Hive CLI


Dears,

Apache Hive 3.1.2 with Hadoop 3.1.1 was working fine with until i configured Hive with Tez based on this doc :[ https://github.com/NitinKumar94/Installing-Apache-Tez]

it always gives this error, i tried many solutions with no luck

     Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/BatchListingOperations
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
    at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3268)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3313)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
    at org.apache.hadoop.hive.common.FileUtils.getJarFilesByPath(FileUtils.java:1006)
    at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5198)
    at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:232)

Any help. i tried the following also with no luck

Copy hive-exec jar from $HIVE_HOME/lib to hdfs dir /user/tez/


Solution

  • If any of below errors and yours you can try this solution

    Error: Could not find or load main class

    org.apache.tez.dag.app.DAGAppMaster

    You need to export your tez confs in hive server instance ( hive env sh)

    export TEZ_CONF_DIR=/etc/tez/conf/
    export TEZ_JARS=/your_path_2_tez/tez-0.10.2/ 
    export HADOOP_CLASSPATH=${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*:${HADOOP_CLASSPATH} 
    

    Then you need to copy your tez folder to hdfs but not zip , you need to copy uncompressed tez package

    hdfs dfs -mkdir /tez/tez-0.10.2
    

    Then copy your files

    hdfs dfs -copyFromLocal /your_path_2_tez/tez-0.10.2/* /tez/tez-0.10.2/
    

    Then define this path in tez-site xml

    <property>
        <name>tez.lib.uris</name>
        <value>${fs.defaultFS}/tez/tez-0.10.2,${fs.defaultFS}/tez/tez-0.10.2/lib</value>
    </property>
    

    Also you need to copy your hadoop version jars (yours 3.1.1) to tez lib folder :

    hadoop-common-3.3.1.jar
    hadoop-yarn-common-3.3.1.jar
    hadoop-yarn-client-3.3.1.jar
    

    ...... .....

    the main idea is which class is not exist looking your exception you can copy the jar from your hadoop share jars from your hadoop installation directory. You need to copy this jars to hdfs as well

    Then you can restart your hive server

    Here you can use tez without error