scalaapache-kafkaapache-zookeeperembedded-kafka

NoClassDefFoundError for embedded-kafka in scala project

I'm trying to integrate an embedded kafka in scala 2.11 for my tests. However I'm getting the following error which I could not understand.

java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback

All the versions of kafka libraries are same. Can you please tell what is wrong in my code?

Here is my dependencies

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
    <version>2.4.7</version>
</dependency>
<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>2.4.0</version>
</dependency>
<dependency>
    <groupId>io.github.embeddedkafka</groupId>
    <artifactId>embedded-kafka_2.11</artifactId>
    <version>2.4.0</version>
    <scope>test</scope>
</dependency>

Here is how I start/stop embedded kafka server.

implicit val embedKafkaConfig = EmbeddedKafkaConfig(kafkaPort = 7000, zooKeeperPort = 7001)
implicit val stringDeserializer = new StringDeserializer()
implicit val stringSerializer = new StringSerializer

Before("") { _: Scenario =>
  EmbeddedKafka.start()(embedKafkaConfig)
}
After("") { _: Scenario =>
  EmbeddedKafka.stop()
}

And the error:

kafka.server.KafkaServer - Fatal error during KafkaServer startup. Prepare to shutdown
java.lang.NoClassDefFoundError: org/apache/zookeeper/AsyncCallback$MultiCallback
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1857)
    at kafka.server.KafkaServer.kafka$server$KafkaServer$$createZkClient$1(KafkaServer.scala:374)
    at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
    at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:52)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:60)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.ops.RunningKafkaOps$class.startKafka(kafkaOps.scala:88)
    at net.manub.embeddedkafka.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:50)
    at net.manub.embeddedkafka.EmbeddedKafka$.start(EmbeddedKafka.scala:68)

enter image description here

Solution

  • The interface org.apache.zookeeper.AsyncCallback.MultiCallback appeared in Zookeeper 3.4.7+

    https://github.com/apache/zookeeper/blob/release-3.4.7/src/java/main/org/apache/zookeeper/AsyncCallback.java#L271

    It was absent in Zookeeper 3.4.6-

    https://github.com/apache/zookeeper/blob/release-3.4.6/src/java/main/org/apache/zookeeper/AsyncCallback.java

    Based on the part of pom.xml you provided, you're using Zookeeper 3.5.6 and this should be ok.

    You should check whether some dependency is still using Zookeeper 3.4.6- in your actual use case.

    I suspect that you provided not enough information for reproduction. I can't reproduce. https://scastie.scala-lang.org/DmytroMitin/q4BRo4taQAey47JufBc87w/1 (see also tab "Build settings"). How to create a Minimal, Reproducible Example. You can check yourself in the following way. Create an empty project with your build file and your code. If you can't reproduce the behavior there then we will not be able either.

    General reasons of NoClassDefFoundError:

    Why am I getting a NoClassDefFoundError in Java?

    What causes and what are the differences between NoClassDefFoundError and ClassNotFoundException?

    NoClassDefFoundError can signal about incompatibilities in dependencies

    Unable to write DF in delta format on hdfs

    ClassNotFoundException: breeze.storage.Zero$DoubleZero$

    NoClassDefFoundError: scala/collection/TraversableOnce (Using Phantom Library To Fetch Data From ScyllaDB Cluster)

    How to fix dependency injection when upgrading play framework from 2.5.x to 2.6.x Scala

    How to get a dependency tree for an artifact?

    Specifying Scalac Compile-Time Option with maven-scala-plugin

    var cl = getClass.getClassLoader
while (cl != null) {
  println(s"classloader: ${cl.getClass.getName}")
  cl match {
    case cl: URLClassLoader =>
      println("classloader urls:")
      cl.getURLs.foreach(println)
    case _ =>
      println("not URLClassLoader")
  }
  cl = cl.getParent
}

    Based on the part of dependency tree you posted, for reproduction it was enough to add

        <dependencies>
        ...
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.7</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

    How can I override zookeper 3.4.6 with 3.5.6 in test scope?

    Try to add to pom.xml

        <dependencies>
        ...
        <dependency>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
            <version>3.5.6</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    How to force a specific version of dependency? (sbt)

    Maven: how to override the dependency added by a library (maven)