The Filebeat in my setup pushes the events to a Kafka cluster with 2 brokers. I have added only one node in the host list but both the brokers in the cluster were discovered. I understood this from the Filebeat logs.
But, even though both the brokers were discovered the events are published to only broker.
Below is teh filebeat config for output
output.kafka:
hosts: ["host1:port"]
topic: '%{[fields.document_type]}'
worker: 4
partition.round_robin:
reachable_only: true
required_acks: 1
compression: gzip
compression_level: 3
Even in the logs it lists that it is connected to only one of the broker as registered one.
2019-01-07T06:12:38.789-0800 INFO kafka/log.go:53 client/metadata fetching metadata for all topics from broker host1:port
2019-01-07T06:12:38.799-0800 INFO kafka/log.go:53 Connected to broker at host1:port (unregistered)
2019-01-07T06:12:38.806-0800 INFO kafka/log.go:53 client/brokers registered new broker #1 at host1:port
2019-01-07T06:12:38.806-0800 INFO kafka/log.go:53 client/brokers registered new broker #0 at host2:port
2019-01-07T06:12:38.806-0800 INFO kafka/log.go:53 kafka message: Successfully initialized new client
2019-01-07T06:12:38.806-0800 INFO pipeline/output.go:105 Connection to kafka(host1:port,host2:port) established
2019-01-07T06:12:38.807-0800 INFO kafka/log.go:53 kafka message: Producer.MaxMessageBytes must be smaller than MaxRequestSize; it will be ignored.
2019-01-07T06:12:38.808-0800 INFO kafka/log.go:53 producer/broker/0 starting up
2019-01-07T06:12:38.814-0800 INFO kafka/log.go:53 Connected to broker at host2:port (registered as #0)
The zookeeper console list both the broker in cluster so the Kafka cluster are fine too.
I am not able to figure the mistake that is causing the Filebeat to push to only one Kafka broker.
Thank you
I think I found the reason for this. My Kafka cluster was on default setting which was set to one partition. Also I started Kafka cluster with one node when the topic was created. Hence all the topic's partition was allocated to one node. When the 2nd node was added Kafka do not automatically re balance the partitions across the nodes causing all partition in one place .Hence Filebeat was sending data to only one node.