cassandracassandra-2.1

Changing commitlog directory in cassandra


Currently commitlog directory is pointing to Directory1. I want to change it different directory D2. How should the migration be ?


Solution

  • This is how we did it. We have a load-balanced client that talks to Cassandra 1.1.2, and each client lives on each Cassandra node.

    1. Drain your service.
      • Wait for the load balancer to remove the node.
    2. Stop your service on the local node to halt direct Client-Cassandra writes: systemctl stop <your service name>
    3. At this point there should be no more writes and greatly reduced disk activity:
      • iostat 2 - Disk activity should be near zero
      • nodetool gossipinfo
    4. Disable Cassandra gossip protocol to mark the node dead and halt Cassandra-Cassandra writes: nodetool disablegossip
    5. Flush all contents of the commit log into SSTables: nodetool flush
    6. Drain the node – this command is more important than nodetool flush, (and might include all the behaviour of nodetool flush): nodetool drain
    7. Stop the cassandra process: systemctl stop cassandra
    8. Modify Cassandra config file(s), e.g. vi /etc/cassandra/default.conf/cassandra.yaml
    9. Start Cassandra: systemctl start cassandra
    10. Wait 10-20 minutes. Tail Cassandra logs to follow along, e.g. tail -F /var/log/cassandra/system.log
    11. Confirm ring is healthy before moving on to next node: nodetool ring
    12. Re-start client service: systemctl start <your service here>

    Note that there was no need for us to do manual copying of the commitlog files themselves. Flushing and draining took care of that. The files then slowly reappeared in the new commitlog_dir location.