kubernetesapache-kafkadebeziumstrimzi

Debezium KafkaConnect, kafka_connect_run.sh No such file or directory


I am getting the error when attempting to setup Debezium in kubernetes (minikube for now) using Strimzi. What is wrong with my manifest? I may be confusing how to use the debezium connect image. I am new to debezium and trying to setup a local test in minikube. Thanks.

Kafka setup via Strimzi:

kubectl create ns kafka
kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
kubectl apply -f https://strimzi.io/examples/latest/kafka/kraft/kafka-single-node.yaml -n kafka

Error:

kafka -n kafka logs my-connect-cluster-connect-0                       
Using BOOTSTRAP_SERVERS=10.99.215.168:9092
Plugins are loaded from /kafka/connect
/docker-entrypoint.sh: line 330: /opt/kafka/kafka_connect_run.sh: No such file or directory

Kubernetes manifest:

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: kafka
spec:
  ports:
    - port: 5432
      targetPort: 5432
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: kafka
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: quay.io/debezium/example-postgres:2.7
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: postgres
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: kafka
spec:
  version: 3.0.0
  replicas: 1
  bootstrapServers: my-cluster-kafka-bootstrap:9092
  image: quay.io/debezium/connect:latest
  resources:
    requests:
      memory: 2Gi
      cpu: 1
    limits:
      memory: 4Gi
      cpu: 2
  config:
    group.id: connect-cluster
    offset.storage.topic: connect-offsets
    config.storage.topic: connect-configs
    status.storage.topic: connect-status
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: inventory-connector
  namespace: kafka
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: io.debezium.connector.postgresql.PostgresConnector
  tasksMax: 1
  config:
    bootstrap.servers: "my-cluster-kafka-bootstrap:9092"
    database.hostname: postgres
    database.port: "5432"
    database.user: postgres
    database.password: postgres
    database.dbname: postgres
    database.server.name: postgres
    slot.name: debezium_slot
    publication.name: debezium_pub
    plugin.name: pgoutput
    topic.prefix: postgres
    snapshot.mode: initial

Solution

  • The container image you use has to be based on the Strimzi Kafka image for the corresponding Strimzi (and Kafka) version. I'm not sure that is the case with quay.io/debezium/connect:latest as the error suggests.

    You can check the Strimzi Docs that contain a guides for how to add connector plugins to the Strimzi image and you have to follow that.