I recently upgraded Elasticsearch and Kibana to v8.9.2 using Bitnami Helm chart on my Kubernetes AWS EKS cluster. Elasticsearch is running fine with 3 nodes but Kibana is restarting again and again as it is failing to start as it is terminating with exit code 1.
Below is the error I'm getting:
FATAL Error: Unable to complete saved object migrations for
the [.kibana_task_manager] index. Error: [{"type":
"unavailable_shards_exception","reason":"[.kibana_task_manager_8.5.1_001][0]
Not enough active copies to meet shard count of [ALL]
(have 1, needed 2). Timeout: [1m], request: [BulkShardRequest
[[.kibana_task_manager_8.5.1_001][0]] containing [26] requests]"}, ...]
I have already checked the following 4 discussions but none of them worked:
What is the actual issue? How to fix it?
I checked my 3 Elasticsearch master nodes pods and containers and all were running fine but when I checked the Elasticsearch cluster health, it was yellow and all of the shards were unassigned_shards.
Then I did some research and found this really good article by Elasticsearch: Elastic Docs › Elasticsearch Guide [8.10] › Troubleshooting › Fix common cluster issues › Red or yellow cluster status
One of the causes were related to Freeing up or increasing disk space which stated:
Elasticsearch uses a low disk watermark to ensure data nodes have enough disk space for incoming shards. By default, Elasticsearch does not allocate shards to nodes using more than 85% of disk space.
So when I looked at the logs of the container of the pod of one of the Elasticsearch master nodes, I found the following: low disk watermark [85%] exceeded on [xxxxxxxxxxxxxxxxxxxxxx][elasticsearch-master-1][/bitnami/elasticsearch/data] free: 196gb[19.9%], replicas will not be assigned to this node
And then after deleting some of the very old unwanted indices, free disk space increased and Elasticsearch started allocating shards and after a few minutes, Kibana was up and running fine and the cluster health changed to green as well.