apache cassandra 3.11.2, 12 nodes, got this error in one node when running nodetool cleanup:
ERROR [CompactionExecutor:4996] 2019-04-10 16:09:43,938 CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:4996,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.compress.CompressionMetadata$Chunk.<init>(CompressionMetadata.java:474) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:239) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:163) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:1137) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:86) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.db.compaction.CompactionManager$6.execute(CompactionManager.java:475) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:322) ~[apache-cassandra-3.11.2.jar:3.11.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_191]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.2.jar:3.11.2]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,613 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6bfdb59e) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1698322065:/data/cassandra/data_files/path/mc-135-big-Index.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,626 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6105deee) to class org.apache.cassandra.io.util.MmappedRegions$Tidier@604462250:/data/cassandra/data_files/path/mc-135-big-Data.db was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,626 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@22504cf1) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@359209204:[Memory@[0..44), Memory@[0..4c8)] was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-04-10 16:10:01,627 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@5ad45619) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1143514085:Memory@[7eaa3bdf4000..7eaa3bdf7200) was not released before the reference was garbage collected
what do we need to do? replace this node with new node? what this error mean? we run the cleanup after nodetool repair successfully completed in all our nodes.
This has been an issue since early versions of Cassandra and has been already reported to the dev team. The condition seems to be innocuous, but if you find out that it only occurs on the same node, you may need to replace it.