I'm reading akka cluster documentation and now I'm at the Gossip section. I did not understand the following statement:
Cluster membership is communicated using a Gossip Protocol, where the current state of the cluster is gossiped randomly through the cluster, with preference to members that have not seen the latest version.
It is pretty hard to imagine. I have the following question:
Question: how does a node know which members have not seen the latest change if the latest change is still gossiped.
I mean, if a node received a notification how it decide where to send it? Obviously, it should not send it back, as the sender have seen the latest change. But what about the other members? Some of them might have already seen the state and there is no any other way to figure it out than query all these members. But if we query one node and it will respond with "no, I have not seen it yet" during the querying someone might send the state to it...
In the next paragraph the article talks about using vector clocks. It says
Convergence is implemented by passing a set of nodes that have seen current state version during gossip.
Note that I have never seen this before in a gossip protocol because it typically converges pretty quickly on its own. I'll go looking for the source, but I recall something about 1000 nodes converging in roughly 10-15 steps when gossipers exchange information instead of just pushing information.