I am trying to set up a 3 node Cassandra VM cluster.
I installed cassandra from datastax package on individual vms and then modified the following:
Seed - vm1 (set the ip address in all the vm configs)
Updated the config with listen_address as the host ip, added the rpc_broadcast_address
Added the cassandra ports in the firewall rules to allow for inter vm communication
Also tried connecting to the vms using SSH
After trying all of this, I started the cassandra seed node, it comes up fine and when I check for the status using nodetool, I see that the node status is UN (Up/Normal). I then start cassandra on the other cluster nodes. It fails to start up with unable to gossip with any seeds error. Any help will be greatly appreciated!
Stack Trace:
java.lang.RuntimeException: Unable to gossip with any seeds
at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1334) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:540) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:783) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:720) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) ~[apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) [apache-cassandra-2.1.9.jar:2.1.9]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:626) [apache-cassandra-2.1.9.jar:2.1.9]
Thanks!
Ok, after spending a day on this, found the issue. It was indeed a firewall issue. In addition to entering the rules in iptables, ensure that the ACCEPT rules appear before the REJECT rules -> ordering matters. After changing the order and restarting the iptables service and cassandra, it started working, the nodes were able to gossip with the seed.