I'm very new to ELK stack and trying to learn about it. I want to understand if how to plan the capacity for 100GB/day data to be kept for 30 days , how to plan for it. The data is mainly access logs, application logs, events. I will need to run few regular expression to populate data fields ( unless there is any default delimiter support for it ) . I'm planning to keep 3 replicas of it .
Can please someone guide me if how many servers and in what type of configuration would be required for this ?
Thanks very much in advance .
We use the "hot-warm architecture" (docs) of clustered elasticsearch instances (you can run multiple on one server!). Each with 31g memory.
Our "hot" instances configured with an retention time of 3 days and an ssd raid. After this three days data is moved to an "warm" HD Raid with an retention time of (30 - 90 days). Its nice because latest data to search in kibana is quite fast processed because of the ssds.
Sometimes we started 1 Elastic Master (4g memory) and 2 Elastic Clients (31g memory) on a single server.
In my opinion such a machine should have at least about 128g memory, 1tb ssd capacity, 4-6tb hd capacity.