hadoophbasehadoop2hbasetestingutility

Upgrading hadoop to 3.1.2 with hbase-testing-utility 2.2.3

The goal

I want to switch from the HDFS to the s3a client. To this end I need to upgrade from Hadoop 2.8.5 to at least 3.1.2, because I need to use the AssumedRoleCredentialProvider for AWS access. Following Table 5 from the HBase docs, Hadoop 3.1.2 should be compatible with HBase 2.2.3.

The issue

I can't get hbase-testing-util to run with the upgraded Hadoop client. It fails to properly start the minicluster.

Code:

  utility = new HBaseTestingUtility(null);
  utility.startMiniCluster();

mvn clean install exception:

java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hdfs.server.namenode.FSDirectory
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:871)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:724)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1103)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
    at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
    at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:759)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:671)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:643)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1096)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1071)

mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2 exception:

java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;
    at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:569)
    at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:550)
    at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:117)
    at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:425)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:160)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1313)
    at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1082)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:957)
    at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:889)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:805)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:671)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:643)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1096)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1071)

Config

I currently run the following functional setup, which breaks when replacing 2.8.5 with 3.1.2

<hbase.version>2.2.3</hbase.version>
<hadoop.version>2.8.5</hadoop.version>
...
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-client</artifactId>
  <version>${hadoop.version}</version>
  <exclusions>
    <exclusion>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.hbase</groupId>
  <artifactId>hbase-client</artifactId>
  <version>${hbase.version}</version>
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>*</artifactId>
    </exclusion>
    <exclusion>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
    </exclusion>
  </exclusions>
</dependency>
<dependency>
  <groupId>org.apache.hbase</groupId>
  <artifactId>hbase-testing-util</artifactId>
  <version>${hbase.version}</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <artifactId>servlet-api</artifactId>
      <groupId>javax.servlet</groupId>
    </exclusion>
    <exclusion>
      <groupId>org.glassfish</groupId>
      <artifactId>javax.el</artifactId>
    </exclusion>
  </exclusions>
</dependency>

What I tried

I already tried excluding / manually including some dependencies with different versions, but to no avail.

Solution

  • The solution is to clone the Apache HBase repo and compile it with hadoop 3

    mvn versions:set -DnewVersion=2.2.3-hadoop3
mvn -Dhadoop.profile=3.0 -Dhadoop-three.version=3.1.2 -DskipTests package deploy