apache-kafkadocker-composeapache-kafka-connect

Broker may not be available in connecting kafka connector with kafka broker in docker compose


I am using docker container to build zookeeper, kafka and connect.

version: '2.1'
services:


  zookeeper:
    image: debezium/zookeeper
    ports:
     - 2181:2181
     - 2888:2888
     - 3888:3888
  kafka:
    image: wurstmeister/kafka
    container_name: kafka-multibinder-1
    ports:
      - "9092:9092"
      - "9094:9094"

    environment:
      - KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

      - KAFKA_ADVERTISED_LISTENERS=INSIDE://:9094,OUTSIDE://localhost:9092
      - KAFKA_LISTENERS=INSIDE://:9094,OUTSIDE://:9092
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      - KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE

    depends_on:
      - zookeeper

  kafka-connect:
    image: debezium/connect
    hostname: kafka-connect
    ports:
     - 8083:8083
    depends_on:
     - kafka
    environment:
     BOOTSTRAP_SERVERS: kafka:9092
     GROUP_ID: 1
     CONFIG_STORAGE_TOPIC: my_connect_configs
     OFFSET_STORAGE_TOPIC: my_connect_offsets

however the logs warms me that 'could not be established. Broker may not be available'

kafka-connect_1  | 2020-10-18 04:11:44,671 INFO   ||  Kafka version: 2.5.0   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,672 INFO   ||  Kafka commitId: 66563e712b0b9f84   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,674 INFO   ||  Kafka startTimeMs: 1602994304669   [org.apache.kafka.common.utils.AppInfoParser]
kafka-connect_1  | 2020-10-18 04:11:44,806 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:44,918 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:11:45,024 WARN   ||  [AdminClient clientId=adminclient-1] Connection to node -1 (/127.0.0.1:9092) could not be established. Broker may not be available.   [org.apache.kafka.clients.NetworkClient]
kafka-connect_1  | 2020-10-18 04:12:44,713 INFO   ||  [AdminClient clientId=adminclient-1] Metadata update failed   [org.apache.kafka.clients.admin.internals.AdminMetadataManager]
kafka-connect_1  | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1602994364719) timed out at 9223372036854775807 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited.
kafka-connect_1  | 2020-10-18 04:12:44,722 ERROR  ||  Stopping due to error   [org.apache.kafka.connect.cli.ConnectDistributed]
kafka-connect_1  | org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:45)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:95)
kafka-connect_1  |  at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:78)
kafka-connect_1  | Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
kafka-connect_1  |  at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
kafka-connect_1  |  at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:58)
kafka-connect_1  |  ... 3 more
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1602994364707) timed out at 1602994364708 after 1 attempt(s)
kafka-connect_1  | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.

Solution

  • In your Kafka configuration port 9094 is used for inter broker communication. But in Kafka connect you trying to connect to the broker via port 9092.

    As you are trying to connect to port 9092 then localhost:9092 will be sent as metadata to the Kafka connect for further communication but localhost:9092 is not available inside the docker network. Use below Kafka connect configuration to connect -

    kafka-connect:
        image: debezium/connect
        hostname: kafka-connect
        ports:
         - 8083:8083
        depends_on:
         - kafka
        environment:
         BOOTSTRAP_SERVERS: kafka:9094
         GROUP_ID: 1
         CONFIG_STORAGE_TOPIC: my_connect_configs
         OFFSET_STORAGE_TOPIC: my_connect_offsets
    

    The concept of Advertised listeners is a little bit trickiest. Refer to this link if you want to know more about listeners and advertised listeners.