azureapache-kafkaapache-kafka-connectdebezium

Error creating CDC Azure Database MySQL to Apache Kafka


Hello guys i wanna ask some question about CDC mysql to kafka with azure database for mysql. I already following this tutorial: https://techcommunity.microsoft.com/t5/azure-database-for-mysql-blog/cdc-in-azure-database-for-mysql-flexible-server-using-kafka/ba-p/2780943

but got stuck while creating kafka conncetor on this part, and the error:

{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nUnable to connect: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}

Kafka connector config:

{
    "name": "sql-server-connection",
    "config": {
        "connector.class": "io.debezium.connector.mysql.MySqlConnector",
        "database.hostname": "localhost",
        "database.port": "3306",
        "database.user": "soleluna",
        "database.dbname": "cdcdatabase",
        "database.password": "mypassword",
        "database.server.id": "1",
        "database.server.name": "userserver",
        "table.whitelist": "dbo.users",
        "database.history": "io.debezium.relational.history.MemoryDatabaseHistory",
        "topic.prefix": "cdc.kafkadev"
    }
}

connect-distributed.properties config:

bootstrap.servers=goldwing.servicebus.windows.net:9093
group.id=connect-cluster-group-1

# connect internal topic names, auto-created if not exists
config.storage.topic=connect-cluster-configs
offset.storage.topic=connect-cluster-offsets
status.storage.topic=connect-cluster-status

# internal topic replication factors - auto 3x replication in Azure Storage
config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

rest.advertised.host.name=connect
offset.flush.interval.ms=10000

key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter

internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

# required EH Kafka security settings
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://goldwing.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=********";

producer.security.protocol=SASL_SSL
producer.sasl.mechanism=PLAIN
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://goldwing.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=********";

consumer.security.protocol=SASL_SSL
consumer.sasl.mechanism=PLAIN
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="Endpoint=sb://goldwing.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=********";

plugin.path=/opt/kafka/libs

The connect-cluster-configs, connect-cluster-offsets, connect-cluster-status topic is already automatically created on azure namespace, can anyone show me where's my mistake? any answer will be appreciate thank you.

My expectation is the debezium kafka connector topic are created and can replicate the data from azure mysql to kafka topic (event hub).


Solution

  • The issue is caused when setting the TLS on the Azure Eventhub.

    Before, I used TLS Version 1.2 (latest). After I change it to TLS Version 1.0, it works fine, no problems, and the default topic (event hub) will automatically be generated on Namespace.