elasticsearchilmindex-management

Elasticsearch ILM terminologies and concepts


I have started reading elasticsearch life cycle management and trying to understand how ILM works. I understood some terminologies like below

Below are something confusing or I don't get it

Before Rollover(Assume a policy configured )

       +--------------+
       |              |
       | metricbeat   |   +----------------+
+------>  (Aliase)    +--->metricbeat7.1   |
       |              |   |                |
       +--------------+   +----------------+

After rollover, the indices are like below(unlink the current index and point to new index)

                                 +--------------------+
                                 | metricbeat7.1      |
                                 | (read only)        |
       +--------------------+    +--------------------+
       |                    |
       |                    |
+----->+  meatricbeat       |     +---------------------+
       |   (Aliase)         |     | metricbeat-0001     |
       |                    +----->  (write index)      |
       +--------------------+     +---------------------+

Am I correct on rollover concept? In blow screen from kibana, There is option "Move to warm phase on rollover" means, move metricbeat7.1 index(from above example) in warm phase? because "rollover happned"? correct?

But what if I unselect "Move to warm phase on rollover", that is still index goes warm phase? right? why do I need that option?

enter image description here

And lastly, what is


Solution

  • Your understanding of the rollover concept is correct. The main idea is that the client keeps indexing data without having to know:

    So what the client actually writes to is an alias that points to a single index. The ILM feature knows when it is time to rollover, i.e. when the existing index has reached the conditions that warrant the creation of a new index. When that's the case, the ILM creates a new index and switches the alias to the new index. All the while, the client keeps writing and doesn't notice anything.

    It is important to know what a hot/warm architecture is. Basically, recent data is kept on (fast, powerful) hot nodes and less recent data is moved to (slower) warm nodes.

    So, when an index is rolled over there are two choices: 1. either the index is kept on hot nodes for a few more days (on the UI you can select the duration before moving the index to the warm nodes) 2. or the index is moved to warm nodes immediately on rollover to make room for the latest index with the most recent data (that's what move to warm phase on rollover means)

    Finally, leader/follower indexes are a different concept more related to cross-cluster replication (CCR) than ILM. A good explanation of what CCR is can be found in this blog article.