mongodbmongodb-cluster

Primary election isn't done after primary is killed on a MongoDB Cluster


I try to test fail over scenario of a mongoDB cluster. When I stopped the primary, I don't see any new primary election on my Java code's logs, and read/write operations are ignore and getting following:

No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=mongo1:30001, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}, ServerDescription{address=mongo2:30002, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3215664, setName='rs0', canonicalAddress=mongo2:30002, hosts=[mongo1:30001], passives=[mongo2:30002, mongo3:30003], arbiters=[], primary='null', tagSet=TagSet{[]}, electionId=null, setVersion=1, lastWriteDate=Fri Mar 26 02:08:27 CET 2021, lastUpdateTimeNanos=91832460163658}, ServerDescription{address=mongo3:30003, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=3283858, setName='rs0', canonicalAddress=mongo3:30003, hosts=[mongo1:30001], passives=[mongo2:30002, mongo3:30003], arbiters=[], primary='null', tagSet=TagSet{[]}, electionId=null, setVersion=1, lastWriteDate=Fri Mar 26 02:08:27 CET 2021, lastUpdateTimeNanos=91832459878686}]}. Waiting for 30000 ms before timing out

I am using the following config:

var cfg = {
    "_id": "rs0",
    "protocolVersion": 1,
    "version": 1,
    "members": [
        {
            "_id": 0,
            "host": "mongo1:30001",
            "priority": 4
        },
        {
            "_id": 1,
            "host": "mongo2:30002",
            "priority": 3
        },
        {
            "_id": 2,
            "host": "mongo3:30003",
            "priority": 2,
        }
    ]
};
rs.initiate(cfg, { force: true });
rs.secondaryOk();
db.getMongo().setReadPref('primary');

rs.isMaster() returns this:

{
    "hosts" : [
        "mongo1:30001"
    ],
    "passives" : [
        "mongo2:30002",
        "mongo3:30003"
    ],
    "setName" : "rs0",
    "setVersion" : 1,
    "ismaster" : true,
    "secondary" : false,
    "primary" : "mongo1:30001",
    "me" : "mongo1:30001",
    "electionId" : ObjectId("7fffffff0000000000000017"),
    "lastWrite" : {
        "opTime" : {
            "ts" : Timestamp(1616719738, 1),
            "t" : NumberLong(23)
        },
        "lastWriteDate" : ISODate("2021-03-26T00:48:58Z"),
        "majorityOpTime" : {
            "ts" : Timestamp(1616719738, 1),
            "t" : NumberLong(23)
        },
        "majorityWriteDate" : ISODate("2021-03-26T00:48:58Z")
    },
    "maxBsonObjectSize" : 16777216,
    "maxMessageSizeBytes" : 48000000,
    "maxWriteBatchSize" : 100000,
    "localTime" : ISODate("2021-03-26T00:49:08.019Z"),
    "logicalSessionTimeoutMinutes" : 30,
    "connectionId" : 28,
    "minWireVersion" : 0,
    "maxWireVersion" : 8,
    "readOnly" : false,
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1616719738, 1),
        "signature" : {
            "hash" : BinData(0,"/+QXGSyYY+M/OXbZ1UixjrDOVz4="),
            "keyId" : NumberLong("6942620613131370499")
        }
    },
    "operationTime" : Timestamp(1616719738, 1)
}

Here what I see is hosts list has primary node and passives list have the secondries. I don't know when is the case that all nodes are considered under hosts in a cluster setup, so passives will be empty. The only related info I found is priority of the secondries should not be 0. Otherwise they won't be considered as candidate for the primary election.

        "mongo1:30001"
    ],
    "passives" : [
        "mongo2:30002",
        "mongo3:30003"
    ],...

Solution

  • From the docs:

    isMaster.passives

    An array of strings in the format of "[hostname]:[port]" listing all members of the replica set which have a members[n].priority of 0.

    This field only appears if there is at least one member with a members[n].priority of 0.

    Those nodes have been set to priority 0 somehow, and will therefore never attempt to become primary.