avroconfluent-schema-registrydebezium

Debezium connect: java.lang.NoClassDefFoundError: com/google/common/base/Ticker

Problem Description

I had trouble using Avro serialization on Debezium and Confluent Schema Registry. I followed the official documentation of Debezium. As it is mentioned in the documentation, from version 2.0 the jar files required to be mounted inside the connect container.

Here is the list of the required jar files:

Workarounds For this purpose, I have downloaded all the required jar files and mounted them to the /kafka/connect/avro_jar_files directory:

Avro required jar files

Debezium version: 2.4

Confluent JAR files version: 7.5.2

Whenever I want to register a new connector (MySQL connector), I face to the following problem:

java.lang.NoClassDefFoundError: com/google/common/base/Ticker
        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:181)
        at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:164)
        at io.confluent.kafka.schemaregistry.client.SchemaRegistryClientFactory.newClient(SchemaRegistryClientFactory.java:36)
        at io.confluent.connect.avro.AvroConverter.configure(AvroConverter.java:71)
        at org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:328)
        at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:620)
        at org.apache.kafka.connect.runtime.Worker.startSourceTask(Worker.java:548)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1833)
        at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$32(DistributedHerder.java:1850)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Ticker
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:136)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)

From the error, I think the com/google/common/base/Tickercom/google/common/base/Ticker class is not included in the kafka-schema-registry-client-7.5.2.jar file.

I didn't find any link that mentioned the compatible version of jar files for the Debezium 2.4 version.

docker-compose.yaml

version: '3'

services:
  zookeeper:
    image: quay.io/debezium/zookeeper:2.4
    container_name: zookeeper
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888

  kafka:
    image: quay.io/debezium/kafka:2.4
    container_name: kafka
    ports:
      - 9092:9092
    environment:
      ZOOKEEPER_CONNECT: zookeeper:2181
    depends_on:
      - zookeeper

  mysql:
    image: quay.io/debezium/example-mysql:2.4
    container_name: mysql
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: debezium
      MYSQL_USER: mysqluser
      MYSQL_PASSWORD: mysqlpw

  mysql_cli:
    image: mysql:8.0
    container_name: mysql_cli
    environment:
      MYSQL_ALLOW_EMPTY_PASSWORD: true
    depends_on:
      - mysql

  connect:
    image: quay.io/debezium/connect:2.4
    container_name: connect
    ports:
      - 8083:8083
    environment:
      GROUP_ID: 1
      # Kafka config
      CONFIG_STORAGE_TOPIC: my_connect_configs
      OFFSET_STORAGE_TOPIC: my_connect_offsets
      STATUS_STORAGE_TOPIC: my_connect_statuses
      BOOTSTRAP_SERVERS: kafka:9092
      # Avro config
      KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
      VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
      CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema_registry:8081
      CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema_registry:8081
      KAFKA_CONNECT_PLUGINS_DIR: /kafka/connect
    volumes:
      - ./jar_files:/kafka/connect/avro_jar_files
    depends_on:
      - kafka
      - schema_registry
      - mysql
  
  schema_registry:
    image: confluentinc/cp-schema-registry:7.1.10 
    container_name: schema_registry
    ports:
      - 8081:8081
    environment:
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
      SCHEMA_REGISTRY_HOST_NAME: schema_registry
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
    depends_on:
      - zookeeper

  kafdrop:
    image: obsidiandynamics/kafdrop:4.0.1
    container_name: kafdrop
    ports:
      - 9000:9000
    environment:
      KAFKA_BROKERCONNECT: kafka:9092
        #      - JVM_OPTS="-Xms32M -Xmx64M"
        #   - SERVER_SERVLET_CONTEXTPATH="/"
    depends_on:
      - kafka

cURL request:

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{
  "name":"inventory-connector",
  "config":{
    "connector.class":"io.debezium.connector.mysql.MySqlConnector",
    "tasks.max":"1",
    "database.hostname":"mysql",
    "database.port":"3306",
    "database.user":"debezium",
    "database.password":"dbz",
    "database.server.id":"184054",
    "topic.prefix":"dbserver1",
    "database.include.list":"inventory",
    "schema.history.internal.kafka.bootstrap.servers":"kafka:9092",
    "schema.history.internal.kafka.topic":"schemahistory.inventory",
    "key.converter":"io.confluent.connect.avro.AvroConverter",
    "value.converter":"io.confluent.connect.avro.AvroConverter",
    "key.converter.schema.registry.url":"http://schema_registry:8081",
    "value.converter.schema.registry.url":"http://schema_registry:8081"
  }
}'
Solution

  • The problem with Debezium and its integration with AVRO on version > 2 is that it is dependent to more libraries (JAR files) that are listed on its official documentation.

    Debezium version: 2.5 (January 2024) Based on its documentation, it is mentioned that the required JAR files are:

    But they aren't sufficient. The problem mentioned in the question is that is has some problem with the Google related libraries (Guava) dependency:

    java.lang.NoClassDefFoundError: com/google/common/base/Ticker

    I have also checked the Confluent link, but it is still not completed!

    After facing to many problem I found that the required JAR files are as below:

    P.S. I have created a repository and placed every scripts and Dockerfile that the problem is fixed on that. For more information, checkout this link.