amazon-web-serviceselasticsearchmagento2amazon-elasticsearch

Updated AWS Elasticsearch storage, but "Minimum free storage space" also increased

I'm a complete newbie with AWS and I'm having issues with AWS Elasticsearch.

We have a large Magento 2 catalog with hundreds of thousands of products (although we only index product name and SKU data). Elasticsearch constantly reboots when indexing is done and kills indexing process.

We have:

Instance type: c5.large.elasticsearch

Number of nodes: 2

EBS volume size: 25GB

From what I understand, it runs into the Minimum Free storage space limit (18GB) and as soon as it hits 18GB per node it reboots the instance.

We have increased the EBS volume size to 30GB, but the Minimum Free Storage space limit also increased to 22GB.

What's the point of increasing the EBS volume size if minimum increases together with it?

Am I missing something? Why did the limit go up?

Here are some graphs with failure points:

Kibana health status

Total free storage space (FreeStorageSpace)

Master instance connection status (MasterReachableFromNode)

UPDATE:

here is the error I'm getting when indexing is done and instance resets:

MasterReachableFromNode-RED

Elasticsearch engine returned an error response. item id: 445689. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445699. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445709. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445719. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445727. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445730. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445732. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
item id: 445739. Error type: "cluster_block_exception", reason "blocked by: [SERVICE_UNAVAILABLE/2/no master];".
Solution

  • Your interpretation of the metric is not right.

    FreeStorageSpace: The free space for data nodes in the cluster. Sum shows total free space for the cluster. Minimum and Maximum show the nodes with the least and most free space, respectively. This metric is also available for individual nodes.

    So when you increase the EBS size, the ability to store more data (FreeStorageSpace) also increases.

    The "cluster_block_exception" error you are getting is due to a different reason - "SERVICE_UNAVAILABLE" - no master. It is recommended to have 3 master eligible nodes for better cluster availability.