I have a MSK cluster using Apache Kafka version 3.7.x
I am attempting to create MSK Sink Connector, connectors in MSK Connect only support Apache Kafka Connect version 2.7.1
I am using self-hosted JDBC Connector plugin downloaded off Confluent: https://www.confluent.io/hub/confluentinc/kafka-connect-jdbc
During creation I get this error:
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,944] WARN [test-aurora-connector-v2|task-0] The configuration 'metrics.context.connect.kafka.cluster.id' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,944] WARN [test-aurora-connector-v2|task-0] The configuration 'metrics.context.connect.group.id' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:369)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,944] INFO [test-aurora-connector-v2|task-0] Kafka version: 2.7.2 (org.apache.kafka.common.utils.AppInfoParser:119)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,944] INFO [test-aurora-connector-v2|task-0] Kafka commitId: unknown (org.apache.kafka.common.utils.AppInfoParser:120)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,945] INFO [test-aurora-connector-v2|task-0] Kafka startTimeMs: 1732802520944 (org.apache.kafka.common.utils.AppInfoParser:121)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,948] INFO [test-aurora-connector-v2|task-0] [Consumer clientId=connector-consumer-test-aurora-connector-v2-0, groupId=connect-test-aurora-connector-v2] Subscribed to topic(s): testsqlfinal.test_db.dbo.test_table (org.apache.kafka.clients.consumer.KafkaConsumer:961)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,949] INFO [test-aurora-connector-v2|task-0] Starting JDBC Sink task (io.confluent.connect.jdbc.sink.JdbcSinkTask:51)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,949] INFO [test-aurora-connector-v2|task-0] JdbcSinkConfig values:
[Worker-05e6001a0fc58e752] auto.create = true
[Worker-05e6001a0fc58e752] auto.evolve = true
[Worker-05e6001a0fc58e752] batch.size = 3000
[Worker-05e6001a0fc58e752] connection.attempts = 3
[Worker-05e6001a0fc58e752] connection.backoff.ms = 10000
[Worker-05e6001a0fc58e752] connection.password = [hidden]
[Worker-05e6001a0fc58e752] connection.url = jdbc:postgresql://test-msk-aurora.cluster-cawjvrsi9ole.eu-west-1.rds.amazonaws.com:5432/test_db?ssl=true&sslmode=require
[Worker-05e6001a0fc58e752] connection.user = postgres
[Worker-05e6001a0fc58e752] date.timezone = DB_TIMEZONE
[Worker-05e6001a0fc58e752] db.timezone = UTC
[Worker-05e6001a0fc58e752] delete.enabled = false
[Worker-05e6001a0fc58e752] dialect.name = PostgreSqlDatabaseDialect
[Worker-05e6001a0fc58e752] fields.whitelist = []
[Worker-05e6001a0fc58e752] insert.mode = insert
[Worker-05e6001a0fc58e752] jdbc.credentials.provider.class = class io.confluent.connect.jdbc.util.DefaultJdbcCredentialsProvider
[Worker-05e6001a0fc58e752] max.retries = 10
[Worker-05e6001a0fc58e752] mssql.use.merge.holdlock = true
[Worker-05e6001a0fc58e752] pk.fields = []
[Worker-05e6001a0fc58e752] pk.mode = none
[Worker-05e6001a0fc58e752] quote.sql.identifiers = ALWAYS
[Worker-05e6001a0fc58e752] retry.backoff.ms = 3000
[Worker-05e6001a0fc58e752] table.name.format = ${topic}
[Worker-05e6001a0fc58e752] table.types = [TABLE]
[Worker-05e6001a0fc58e752] trim.sensitive.log = false
[Worker-05e6001a0fc58e752] (io.confluent.connect.jdbc.sink.JdbcSinkConfig:361)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,949] INFO [test-aurora-connector-v2|task-0] Initializing JDBC writer (io.confluent.connect.jdbc.sink.JdbcSinkTask:65)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,949] INFO [test-aurora-connector-v2|task-0] Validating JDBC URL. (io.confluent.connect.jdbc.dialect.DatabaseDialects:171)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,949] INFO [test-aurora-connector-v2|task-0] Validated JDBC URL. (io.confluent.connect.jdbc.dialect.DatabaseDialects:174)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,949] ERROR [test-aurora-connector-v2|task-0] WorkerSinkTask{id=test-aurora-connector-v2-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:193)
[Worker-05e6001a0fc58e752] java.lang.NoSuchMethodError: 'boolean org.apache.kafka.common.utils.Utils.isBlank(java.lang.String)'
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.util.StringUtils.isNotBlank(StringUtils.java:79)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.getJdbcCredentialsProvider(GenericDatabaseDialect.java:2010)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.dialect.GenericDatabaseDialect.<init>(GenericDatabaseDialect.java:192)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.dialect.PostgreSqlDatabaseDialect.<init>(PostgreSqlDatabaseDialect.java:99)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.dialect.PostgreSqlDatabaseDialect$Provider.create(PostgreSqlDatabaseDialect.java:74)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.dialect.DatabaseDialects.create(DatabaseDialects.java:156)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.sink.JdbcSinkTask.initWriter(JdbcSinkTask.java:67)
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.sink.JdbcSinkTask.start(JdbcSinkTask.java:53)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:309)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:240)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[Worker-05e6001a0fc58e752] at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,950] INFO [test-aurora-connector-v2|task-0] Stopping task (io.confluent.connect.jdbc.sink.JdbcSinkTask:178)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,950] WARN [test-aurora-connector-v2|task-0] Could not stop task (org.apache.kafka.connect.runtime.WorkerSinkTask:171)
[Worker-05e6001a0fc58e752] java.lang.NullPointerException
[Worker-05e6001a0fc58e752] at io.confluent.connect.jdbc.sink.JdbcSinkTask.stop(JdbcSinkTask.java:180)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerSinkTask.close(WorkerSinkTask.java:169)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerTask.doClose(WorkerTask.java:170)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:196)
[Worker-05e6001a0fc58e752] at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:240)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[Worker-05e6001a0fc58e752] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[Worker-05e6001a0fc58e752] at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,951] INFO [test-aurora-connector-v2|task-0] Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:668)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,951] INFO [test-aurora-connector-v2|task-0] Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:672)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,951] INFO [test-aurora-connector-v2|task-0] Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:678)
[Worker-05e6001a0fc58e752] [2024-11-28 14:02:00,951] INFO [test-aurora-connector-v2|task-0] App info kafka.consumer for connector-consumer-test-aurora-connector-v2-0 unregistered (org.apache.kafka.common.utils.AppInfoParser:83)
I don't know what I can do, is it AWS MSK runtime that's outdated? Is it the plugin JAR versions that are incorrect? Does anyone know how to solve this problem?
Not outdated; Too new.
Yes - this error is a sign that you must use a lesser plugin version that's been compiled against kafka 2.7.1
Alternatively, run Kafka Connect 3.7.x in ECS, EC2, for example, on your own