elasticsearchloggingprometheusfluentdobservability

fluentd vs prometheus vs elasticsearch, when to use what?


When to use fluentd vs prometheus vs elasticsearch? There seems to be overlapping between these tools interms of the feature set they provide but I have seen projects using all these 3 tools in unison. Quite confusion why these 3 tools has to be in the same solution. Does these 3 tools need to be used together? as it may requires different management team and expertise.

fluentd - for application log management not metrics - collects log, do transformation and ingest to various destinations

prometheus -for scraping metrics especially on container environment. Does this do application log management as well? not sure

elastricsearch - scales well for log storage & do efficient search queries.

Does I missed anything form above understanding? need your expert opinions.


Solution

  • Fluentd: Log collection and forwarding.

    Prometheus: Metrics collection and alerting.

    Elasticsearch: Log storage, search, and visualization.

    While there's an overlap, Fluentd, Prometheus, and Elasticsearch each serve different primary purposes. However, in a complex environment, especially in the context of microservices or container orchestration systems like Kubernetes, having a comprehensive view through metrics (Prometheus) and logs (Fluentd + Elasticsearch) is valuable. That said, teams should evaluate their actual needs and the associated overhead before adopting all three simultaneously.