[SOLVED] Apache Ignite - Automatic Cluster Snapshot Create / Restore Procedure

When trying the Apache Ignite cluster snapshot procedure (snapshot create, restore), the restore operation is not successful.

Before the restore procedure, we have destroyed the cache which we want to restore (according to the official documentation), but still the same error message / Java Exception is present in the logs.

Apache Ignite - Snapshot Create / Restore Procedure:

./bin/control.sh --snapshot create 20230412_snapshot_test_v1
./bin/control.sh --cache destroy --caches CACHE_XYZ --yes
./bin/control.sh --snapshot restore 20230412_snapshot_test_v1 --start --groups CACHE_XYZ

Apache Ignite - Logs:

The snapshot operation will be aborted due to a handler error 
class org.apache.ignite.IgniteCheckedException: The check procedure failed on 3 nodes.

The check procedure failed on nodes:

Node ID: 260115a7-95b4-48a3-bc98-af9d8af0df51 [192.168.XX.YY, 127.0.0.1]
Consistent ID: 547359f6-4501-4727-8563-a8b170a905f3
Exception: java.lang.NegativeArraySizeException
-935925367

Node ID: 33d84ca7-e4e2-4b6e-9c3c-63625c853dc1 [192.168.XX.YY, 127.0.0.1]
Consistent ID: aa7da35d-37b3-401d-8df3-f3c6e2761c85
Exception: java.lang.NegativeArraySizeException
-935925367

Node ID: d512f146-abb2-4e88-a22c-3a640e84be18 [192.168.XX.YY, 127.0.0.1]
Consistent ID: 0b6fafa5-a31f-4fa7-80c5-44c792af03ff
Exception: java.lang.NegativeArraySizeException
-935925367

...

Failed to restore snapshot cache groups 
class org.apache.ignite.compute.ComputeUserUndeclaredException: Failed to reduce job results due to undeclared user exception [task=org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotHandlerRestoreTask@4c606da9, err=class org.apache.ignite.IgniteException: The check procedure failed on 3 nodes.

Another weird point, for some reason the --snapshot check is not able to connect to the cluster, but all other commands work normally.

Apache Ignite - Cluster Snapshot Verification:

./bin/control.sh --snapshot check 20230412_snapshot_test_v1 --verbose

...

INFO: Client TCP connection established: /127.0.0.1:11211

...

SEVERE: Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker [readBuf=java.nio.DirectByteBuffer[pos=0 lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-client-0, igniteInstanceName=gridClient, finished=false, heartbeatTs=1682407593249, hashCode=245934772, interrupted=false, runner=grid-nio-worker-tcp-client-0-#5%gridClient%]]], writeBuf=null, readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true, outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:42122, rmtAddr=/127.0.0.1:11211, createTime=1682407593249, closeTime=0, bytesSent=702, bytesRcvd=78929, bytesSent0=702, bytesRcvd0=78929, sndSchedTime=1682407593249, lastSndTime=1682407593249, lastRcvTime=1682407593249, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser [marsh=JdkMarshaller [clsFilter=null], routerClient=false], directMode=false]], accepted=false, markedForClose=false]]]
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1183)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2508)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
        at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1910)
        at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
        at java.lang.Thread.run(Thread.java:750)

...

class org.apache.ignite.internal.client.impl.connection.GridClientConnectionResetException: Failed to perform request (connection failed): /127.0.0.1:11211

Apache Ignite - Logs:

[SnapshotPartitionsVerifyTask$VisorVerifySnapshotPartitionsJob] Verify snapshot partitions procedure has been initiated [snpName=20230412_snapshot_test_v1, consId=33303d92-f3db-4b33-be38-798167c2f726]
[IgniteSnapshotManager] Resolved directory for serialized binary metadata: /usr/share/apache-ignite/work/snapshots/20230412_snapshot_test_v1/db/binary_meta/node00-33303d92-f3db-4b33-be38-798167c2f726

... nothing else is printed, the service goes down ...

After the snapshot verification fails (connection error explained above), the Apache Ignite service is immediately stopped on every Linux server, the cluster goes down and everything needs to be started again manually.

It's a three-node Apache Ignite cluster (each node is on a different server) with the Apache ZooKeeper configured, and we have the same issue on Apache Ignite 2.13.0 and 2.14.0, running on Linux with OpenJDK 1.8.0.

What could be the reason, why is the restore procedure failing ?