Suppose nodes A, B, C, and D are clustered by the following configuration,
cluster {
seed-nodes = ["A", "B", "C", "D"]
}
and nodes C and D with the addition of nodes E, F, and G are soon to be deployed under the following configuration:
cluster {
seed-nodes = ["C", "D", "E", "F", "G"]
}
In this case, would the order of deployment affects the cluster formation? In other words, would the two scenarios below result in the same cluster:
- deploy E, F, and G, then re-deploy C and D, or
- re-deploy C and D, then deploy E, F and G.
As per the Akka documentation, the first node in seed-nodes plays a special role in cluster formation:
but the node configured as the first element in the seed-nodes list must be started when initially starting a cluster. If it is not, the other seed-nodes will not become initialized, and no other node can join the cluster.
Since node C in the latter case starts with no nodes D, E, F, and G belong to a cluster, it should create a new cluster. But, in the former case, before C and D are re-deployed, nodes E, F, and G joins the existing cluster via C and D, and the re-deployment C and D would not change anything since there's some nodes already belong to a cluster. Am I right?
Maybe, depending on how you "redeploy". But let me step through it in detail, as I had to read your question multiple times to (I think) understand what you are asking. Then I'll make some general recommendations.
Essentially you have two configurations: one with [A,B,C,D] as seed nodes and another with [C,D,E,F,G] as seed nodes. I'll refer to these as config 1 and config 2. You then have two scenarios where you are starting and stopping and reconfiguring the nodes in various orders. But I wasn't exactly sure about one of your steps so I actually define scenario 1, scenario 2a, and scenario 2b.:
- Scenario 1: Start nodes in the order A-1, B-2, C-1, D-1. (Meaning start those nodes with config 1.) Wait for cluster formation. Deploy E-2, F-2, G-2. Stop C-1, D-1, reconfigure, deploy C-2 then D-2.
- Scenario 2a: Start nodes in the order A-1, B-2, C-1, D-1. Wait for cluster formation. Stop C-1, reconfigure C, then deploy C-2, then top D-1, reconfigurate D, deploy D-2, then E-2, F-2, G-2.
- Scenario 2b: Start nodes in the order A-1, B-2, C-1, D-1. Wait for cluster formation. Stop C-1, stop D-1. reconfigure C+D,, then deploy C-2, deploy D-2, then E-2, F-2, G-2. (The difference from 2a being that you stop both C and D at the same time.)
As you point out, the first node in the seeds has the unique ability to start a new cluster. So if we walk through scenario 1:
- A-1 starts. He attempts to discover a running cluster from B,C, or D. He can't find a running cluster in any of those places, so he will start a new cluster using his special ability.
- B-1 then starts, and attempts to find a cluster from A,C, or D. He finds one at A and joins it.
- C-1 then starts, and attempts to find a cluster from A,B, or D. He might find it at either A or B, it doesn't matter.
- D-1 then starts, follows the same process finds the cluster and joins the cluster.
- You don't really specify that they are started in this order, but it doesn't matter. Since they have the same config they will all eventually form a cluster of four nodes.
- E-2 then deploys. Even though he has a different config, it doesn't really matter: he will be looking for a cluster from C,D,F,G. And he will find it at either C or D and will join.
- Same for F and G.
- You then say "re-deploy C and D", which I will interpret as stopping, changing the config, and restarting. Assuming that E, F, and G, came up successfully, C and D will find the cluster through one of those three servers and join. Through joining the cluster they will learn about A and B; the seed nodes are just for cluster discovery. Once joined they get the complete list of nodes from the consensus.
- So we end up with one cluster with all nodes: A,B,C,D,E,F,G.
Scenario 2a:
- Same initial process for creating the initial cluster of A,B,C,D.
- But then C, and D are "re-deployed". If you stop, reconfigure, and restart them one at a time then C will find the existing cluster through D. Then D will find the cluster through the restarted C.
- Then when you start E, F, and G they will find the cluster through any of the others and you will end up with one cluster with all nodes: A,B,C,D,E,F,G.
Scenario 2b:
- Same initial process for creating the initial cluster of A,B,C,D.
- But then C, and D are "re-deployed". This time we assume that you shut them both down. Reconfigure them both. Then redeploy them both.
- In this case when C is restarted his configuration includes no mention of A or B, so he has no way to find them (and thus no way to find the original cluster). So he, since he is now listed as the initial seed node, will use his unique ability to start a new cluster.
- D will then join the new cluster (since he similarly has no way to find the original). And then the same with E,F,G and so forth since they will easily be able to find this second cluster. (But will have no way to find the initial cluster.)
- You will end up with two clusters: one with A,B and a second with C,D,E,F,G.
Several comments about this:
- This shows why seed nodes aren't really the preferred strategy.
- It shows why the first entry in the seed node list is important: the ability to create a new cluster is restricted specifically to prevent orphan clusters like this. When you change it, you do need to make sure than when this new "special node" starts that it isn't going to start a new cluster. (Unless that's specifically what you are aiming for.)
- Your example lists all nodes as seed nodes. That's somewhat unusual. You aren't listing all of the nodes; you are just giving new nodes a "starting point" so that they can find existing clusters. And argument could be made that you should really only have 2-3 seed nodes no matter how big your cluster. In part because you don't want to have to be changing that list anytime you want to add members. (Because then you end up in scenarios like this.)
- This also is sort of showing that the intent is for clusters to be very long running. I actually knew of one case where they deliberately stopped the initial seed node once the cluster was formed so that it was impossible for new clusters to be created. (Only restarting that node manually in the very rare cases when they wanted restart a cluster.)