Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).
I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode, possibly hacked?
When trying to get Kibana running, I get the following:
Tried restarting Kibana - it restarted, nothing changed. Tried restarting Elastic - it restarted (all nodes), nothing changed.
I then had a look at the cluster settings and this is what I got:
{
"persistent": {
"cluster": {
"routing": {
"allocation": {
"enable": "all"
}
},
"blocks": {
"read_only": "true"
}
}
},
"transient": {
"cluster": {
"routing": {
"allocation": {
"enable": "all"
}
}
}
}
}
I tried undoing the read only as follows:
PUT _cluster/settings
{
"persistent": {
"blocks.read_only": false
}
}
No luck as you can see:
{
"error": {
"root_cause": [
{
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
}
],
"type": "cluster_block_exception",
"reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
},
"status": 403
}
Any ideas?
UPDATE: Problem solved by Andrei Stefan, now for the more important part - why? What happened and why? I've lost all data and my cluster entered a read-only mode.
The correct command is:
PUT /_cluster/settings
{
"persistent" : {
"cluster.blocks.read_only" : false
}
}