cassandracluster-computingcassandra-2.0gossip

Cassandra Cluster Set up - Unable to gossip with any seeds


I am trying to set up a 3 node Cassandra VM cluster.

I installed cassandra from datastax package on individual vms and then modified the following:

  1. Seed - vm1 (set the ip address in all the vm configs)

  2. Updated the config with listen_address as the host ip, added the rpc_broadcast_address

  3. Added the cassandra ports in the firewall rules to allow for inter vm communication

  4. Also tried connecting to the vms using SSH

After trying all of this, I started the cassandra seed node, it comes up fine and when I check for the status using nodetool, I see that the node status is UN (Up/Normal). I then start cassandra on the other cluster nodes. It fails to start up with unable to gossip with any seeds error. Any help will be greatly appreciated!

Stack Trace:

java.lang.RuntimeException: Unable to gossip with any seeds
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1334) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:540) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:783) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:720) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) [apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:626) [apache-cassandra-2.1.9.jar:2.1.9]

Thanks!


Solution

  • Ok, after spending a day on this, found the issue. It was indeed a firewall issue. In addition to entering the rules in iptables, ensure that the ACCEPT rules appear before the REJECT rules -> ordering matters. After changing the order and restarting the iptables service and cassandra, it started working, the nodes were able to gossip with the seed.