aws-elasticsearchelasticsearch-opendistro

AWS Elasticsearch OpenDistro - Rollover indices not triggered


I have set up a 1-node cluster using AWS Elasticsearch service v7.10 and I've followed this OpenDistro guides to create simple index rollover policy, however the policy did not trigger at all even I waited for hours. Below is my ISM policy:

PUT _opendistro/_ism/policies/book_rollover_policy
{
    "policy": {
        "policy_id": "book_rollover_policy",
        "description": "Book rollover policy.",
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "replica_count": {
                            "number_of_replicas": 0
                        },
                        "rollover": {
                            "min_index_age": "3m"
                        }
                    }
                ],
                "transitions": []                
            }            
        ],
        "ism_template": {
            "index_patterns": [
                "book-*"
            ],
            "priority": 100
        }
    }
}

This is the template with rollover alias:

PUT _template/book_ism_rollover
{
  "index_patterns": [
    "book-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "opendistro": {
        "index_state_management": {
          "rollover_alias": "book"
        }
      },
      "analysis": {}
    }
  },
  "mappings": {
    "properties": {
      "author": {
        "type": "text"
      },
      "isbn": {
        "type": "text"
      },
      "price": {
        "type": "float"
      },
      "publishedDate": {
        "type": "date"
      },
      "publisher": {
        "type": "text"
      },
      "title": {
        "type": "text"
      }
    }
  }
}

Then I created initial rollover index with alias:

PUT book-000001
{
  "aliases": {
    "book": {
      "is_write_index": true
    }
  }
}

Then I started indexing a few documents as below:

#Book 1
POST book/_doc
{
  "isbn": "f2338e3e-fabf-45bb-aa6e-5f8cb6f259c1",
  "author": "Jon Skeet",
  "title": "C# in depth",
  "publiser": "Manning Publications",
  "publishedDate": "2008-01-21",
  "price": 29.99
}
#Book 2
POST book/_doc
{
  "isbn": "30e57ff1-98f9-405b-aede-5df460455a5d",
  "author": "Joseph Albahari",
  "title": "C# 9.0 in the nutshell",
  "publiser": "O'Reilley",
  "publishedDate": "2018-03-01",
  "price": 70.99
}
#Book 3
POST book/_doc
{
  "isbn": "14fa6134-f9e4-4525-9d28-f3def875c35c",
  "author": "Mark Lutz",
  "title": "Learning Python",
  "publiser": "O'Reilley",
  "publishedDate": "2010-12-01",
  "price": 30.99
}

With the policy above I expect a new index would be created after 3 minutes but nothing happened. I also tried different rollover actions like min_size and min_doc_count but still no luck. Am I missing anything? Thanks.


Solution

  • I've finally figured out with the following changes, not sure if I had to have all of them or just a few of them.

    1. I made a 2-node cluster instead of 1-node cluster and downgraded the Elasticsearch version from 7.10 to 7.9
    2. I ran this request PUT /_cluster/settings { "persistent": { "opendistro.index_state_management.enabled": true } }
    3. Lastly, from this I had to wait 30-48 minutes to see the new index got rolled over.