During cassandra decommission I noticed that the node tries to send hints which takes extremely long time and never completes. I checked the hints folder and found hints that are more than 9+ months old. I am not sure why those old hints are still present in the folder so I decided to delete them. After I deleted them I noticed the following entry in the system.log
INFO [HintsDispatcher:1070] 2021-07-08 11:32:01,056 HintsDispatchExecutor.java:141 - Transferring all hints to /10.199.190.233: 7935f1b5-4725-4dc2-ad6d-b883d53d907d
ERROR [HintsDispatcher:1070] 2021-07-08 11:32:01,061 CassandraDaemon.java:207 - Exception in thread Thread[HintsDispatcher:1070,1,RMI Runtime]
java.lang.RuntimeException: java.nio.file.NoSuchFileException: /data/cassandra/data/hints/ce6bb0e3-849f-487d-9274-38b8536b89cf-1603947885707-1.hints
Where does Cassandra keep metadata for the hints as system.hints folder didn't have any entry?
Cassandra Version is 3.0.12
There is a catalog of hints held in memory on each Cassandra node for tracking.
If you manually delete the contents of the hints directory on a node, the entries in the hints catalog become stale and you run into the NoSuchFileException you posted.
The correct way of deleting hints is with the nodetool truncatehints command. Cheers!