We're running Cassandra 2.1.14 and migrated to TWCS on one of our CFs a few months ago and since then, while getting better performance and avg SSTables read per query reduced drastically, we noticed old SSTables never get evicted.
Running Cassandra's sstableexpiredblockers utility we see the reason is old SSTabes's max timestamp are higher than even new SSTable's min timestamp, which means Cassandra won't try to evict even though the actual SSTables are completely tombstoned (we wrote a tiny java app that went through the SSTable directly to make sure).
So the question is - why do even very new SSTables have old rows? Just to clarify: our code base NEVER update rows, and ALL rows have TTL.
Another question, given that this is our current situation, is there an easy way to force Cassandra to run eviction on old SSTables?
Attached is the SSTables and their min \ max timestamp:
So we found the root cause of the issue is read repair and speculative retry were enabled. disabling them and running major compaction once to delete the stale SSTables and everything worked as expected.