Problem Description
I had trouble using Avro serialization on Debezium
and Confluent Schema Registry
. I followed the official documentation of Debezium. As it is mentioned in the documentation, from version 2.0
the jar files required to be mounted inside the connect container.
Here is the list of the required jar files:
Workarounds
For this purpose, I have downloaded all the required jar files and mounted them to the /kafka/connect/avro_jar_files
directory:
Debezium version: 2.4
Confluent JAR files version: 7.5.2
Whenever I want to register a new connector (MySQL connector), I face to the following problem:
java.lang.NoClassDefFoundError: com/google/common/base/Ticker
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:181)
at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.<init>(CachedSchemaRegistryClient.java:164)
at io.confluent.kafka.schemaregistry.client.SchemaRegistryClientFactory.newClient(SchemaRegistryClientFactory.java:36)
at io.confluent.connect.avro.AvroConverter.configure(AvroConverter.java:71)
at org.apache.kafka.connect.runtime.isolation.Plugins.newConverter(Plugins.java:328)
at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:620)
at org.apache.kafka.connect.runtime.Worker.startSourceTask(Worker.java:548)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1833)
at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$getTaskStartingCallable$32(DistributedHerder.java:1850)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: com.google.common.base.Ticker
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:136)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
From the error, I think the com/google/common/base/Tickercom/google/common/base/Ticker
class is not included in the kafka-schema-registry-client-7.5.2.jar
file.
I didn't find any link that mentioned the compatible version of jar files for the Debezium 2.4 version.
docker-compose.yaml
version: '3'
services:
zookeeper:
image: quay.io/debezium/zookeeper:2.4
container_name: zookeeper
ports:
- 2181:2181
- 2888:2888
- 3888:3888
kafka:
image: quay.io/debezium/kafka:2.4
container_name: kafka
ports:
- 9092:9092
environment:
ZOOKEEPER_CONNECT: zookeeper:2181
depends_on:
- zookeeper
mysql:
image: quay.io/debezium/example-mysql:2.4
container_name: mysql
ports:
- 3306:3306
environment:
MYSQL_ROOT_PASSWORD: debezium
MYSQL_USER: mysqluser
MYSQL_PASSWORD: mysqlpw
mysql_cli:
image: mysql:8.0
container_name: mysql_cli
environment:
MYSQL_ALLOW_EMPTY_PASSWORD: true
depends_on:
- mysql
connect:
image: quay.io/debezium/connect:2.4
container_name: connect
ports:
- 8083:8083
environment:
GROUP_ID: 1
# Kafka config
CONFIG_STORAGE_TOPIC: my_connect_configs
OFFSET_STORAGE_TOPIC: my_connect_offsets
STATUS_STORAGE_TOPIC: my_connect_statuses
BOOTSTRAP_SERVERS: kafka:9092
# Avro config
KEY_CONVERTER: io.confluent.connect.avro.AvroConverter
VALUE_CONVERTER: io.confluent.connect.avro.AvroConverter
CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL: http://schema_registry:8081
CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL: http://schema_registry:8081
KAFKA_CONNECT_PLUGINS_DIR: /kafka/connect
volumes:
- ./jar_files:/kafka/connect/avro_jar_files
depends_on:
- kafka
- schema_registry
- mysql
schema_registry:
image: confluentinc/cp-schema-registry:7.1.10
container_name: schema_registry
ports:
- 8081:8081
environment:
SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
SCHEMA_REGISTRY_HOST_NAME: schema_registry
SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081
depends_on:
- zookeeper
kafdrop:
image: obsidiandynamics/kafdrop:4.0.1
container_name: kafdrop
ports:
- 9000:9000
environment:
KAFKA_BROKERCONNECT: kafka:9092
# - JVM_OPTS="-Xms32M -Xmx64M"
# - SERVER_SERVLET_CONTEXTPATH="/"
depends_on:
- kafka
cURL request:
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{
"name":"inventory-connector",
"config":{
"connector.class":"io.debezium.connector.mysql.MySqlConnector",
"tasks.max":"1",
"database.hostname":"mysql",
"database.port":"3306",
"database.user":"debezium",
"database.password":"dbz",
"database.server.id":"184054",
"topic.prefix":"dbserver1",
"database.include.list":"inventory",
"schema.history.internal.kafka.bootstrap.servers":"kafka:9092",
"schema.history.internal.kafka.topic":"schemahistory.inventory",
"key.converter":"io.confluent.connect.avro.AvroConverter",
"value.converter":"io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url":"http://schema_registry:8081",
"value.converter.schema.registry.url":"http://schema_registry:8081"
}
}'
The problem with Debezium and its integration with AVRO on version > 2 is that it is dependent to more libraries (JAR files) that are listed on its official documentation.
Debezium version: 2.5 (January 2024) Based on its documentation, it is mentioned that the required JAR files are:
But they aren't sufficient. The problem mentioned in the question is that is has some problem with the Google related libraries (Guava) dependency:
java.lang.NoClassDefFoundError: com/google/common/base/Ticker
I have also checked the Confluent link, but it is still not completed!
After facing to many problem I found that the required JAR files are as below:
P.S. I have created a repository and placed every scripts and Dockerfile that the problem is fixed on that. For more information, checkout this link.