Hey folks can anyone explain to me why does i need to use fluentd for aggregator? Below is the infrastructure i have been working on
Fluentbit(Log Forwarder) --> Fluentd(Data Processing and Aggregator) --> Elasticsearch --> Kibana
Sorry for not show picture, but you can find the topologhy what i mean in here : https://medium.com/redbox-techblog/building-an-open-data-platform-logging-with-fluentd-and-elasticsearch-4582de868398
Okay maybe this is a weird a question, but i still dont understand why i need it? when i just can use the fluentbit to forward the log straight to elasticsearch, why we must need the aggregator?
For more information i will be using a lot of fluentbit for couple of node(around 50 nodes i think)
I appriciate if you help me, and sorry for my bad english
Thank you
You don't need it, just make sure your Elasticsearch cluster is properly setup so it can handle the load with several incoming connections and traffic.