akkacluster-computingsplitbrain

Detecting killed nodes with Akka Cluster


I'm working on a project using Akka Cluster 2.4.8.

Is there a way to detect crashed nodes (as in computer failure, kill -9, etc) with AkkCluster ?

I'm currently have a 3-node environment, using the static-quorum split brain resolve strategy.

akka.cluster.split-brain-resolver {
    active-strategy = static-quorum
    stable-after = 5s

    static-quorum {
        quorum-size = 2
        role = ""
    } 

I was hoping that when killing an instance, the remaining cluster members would mark it as DOWN. However, it remains UNREACHABLE (see below). Is there a way to achieve this?

clusterStatus": {
    "members": [
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-00",
                    "port": 2552
                },
                "uid": 1998600863
            },
            "upNumber": 1,
            "status": "Up",
            "roles": []
        },
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-01",
                    "port": 2552
                },
                "uid": 1371217592
            },
            "upNumber": 3,
            "status": "Up",
            "roles": []
        },
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-02",
                    "port": 2552
                },
                "uid": -796176254
            },
            "upNumber": 2,
            "status": "Up",
            "roles": []
        }
    ],
    "unreachable": [
        {
            "uniqueAddress": {
                "address": {
                    "protocol": "akka.tcp",
                    "system": "test-actor-system",
                    "host": "test-out-01",
                    "port": 2552
                },
                "uid": 1371217592
            },
            "upNumber": 3,
            "status": "Up",
            "roles": []
        }
    ]

Solution

  • Split-brain resolver is a commercial feature of Akka for which you need a Lightbend subscription.

    Note

    This is a feature of the Typesafe Reactive Platform that is exclusively available for Typesafe Project Success Subscription customers.

    To use the Split Brain Resolver feature you must install Typesafe Reactive Platform.

    Chances are, if you're not a Reactive Platform subscriber, your split-brain configs are just plainly ignored.

    Full docs at http://doc.akka.io/docs/akka/rp-15v09p02/scala/split-brain-resolver.html