elasticsearchapache-kafkalogstash

Could not connect Logstash to Kafka via compose file


I'm using compose file to create data pipeline between Logstash and Kafka. But this message shows up in logstash container. Could someone help me out?

The message:

[WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Connection to node 2 could not be established. Broker may not be available.

My compose file:

version: "3"
services:
    zookeeper:
        image: confluentinc/cp-zookeeper:6.2.0
        container_name: zookeeper
        ports:
            - "2181:2181"
        networks:
            - kafkanet
        environment:
            ZOOKEEPER_CLIENT_PORT: "2181"
            ZOOKEEPER_TICK_TIME: "2000"
            ZOOKEEPER_SYNC_LIMIT: "2"
        
    kafkaserver:
        image: confluentinc/cp-kafka:6.2.0
        container_name: kafka
        ports:
            - "9092:9092"
        networks:
            - kafkanet
        environment:
            KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
            KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
            KAFKA_BROKER_ID: "2"
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
        depends_on:
            - zookeeper

    elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:6.4.0
        container_name: elasticsearch
        ports:
            - 9200:9200
            - 9300:9300
        networks:
            - kafkanet
    
    kibana:
        image: docker.elastic.co/kibana/kibana:6.4.0
        container_name: kibana
        ports:
            - 5601:5601
        networks:
            - kafkanet
        depends_on: [ 'elasticsearch' ]
    
    # Logstash Docker Image
    logstash:
        image: docker.elastic.co/logstash/logstash:6.4.0
        container_name: logstash
        networks:
            - kafkanet
        depends_on: [ 'elasticsearch', 'kafkaserver' ]
        volumes:
            - './logstash/config:/usr/share/logstash/pipeline/'

networks:
    kafkanet:
        driver: bridge

./logstash/config/logstash.conf

input {
  kafka {
    bootstrap_servers => "kafkaserver:9092"
    topics => ["sit.catalogue.item","uat.catalogue.item"]
    auto_offset_reset => "earliest"
    decorate_events => true
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "%{[indexPrefix]}-logs-%{+YYYY.MM.dd}"
  }
}


Solution

  • Your advertised listener in Kafka is not right. It should be kafkaserver.

    So instead of

    KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://localhost:9092"
    

    You need

    KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafkaserver:9092"
    

    For more details, see this blog that I wrote.