mongodbmongodb-indexes

MongoDB stuck creating index with "Index Build: draining writes received during build" message


I have 4 rows in the test collection:

{ "_id" : ObjectId("5f4ce50e19b13337216dd477"), "test" : 1 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd478"), "test" : 2 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd479"), "test" : 3 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd47a"), "test" : 4 }

After running db.test.createIndex({test:1},{background:1}); to create an index, it just hangs. It was hanging for at least a few hours. Here is what I found in the db.currentOp() about this operation:

{
    "type" : "op",
    "host" : "HOSTNAME:27017",
    "desc" : "IndexBuildsCoordinatorMongod-13",
    "active" : true,
    "currentOpTime" : "2020-08-31T12:11:13.159+00:00",
    "opid" : 8721867,
    "secs_running" : NumberLong(20),
    "microsecs_running" : NumberLong(20888590),
    "op" : "command",
    "ns" : "test.test",
    "command" : {
        "createIndexes" : "test",
        "indexes" : [
            {
                "v" : 2,
                "key" : {
                    "test" : 1
                },
                "name" : "test_1",
                "background" : 1
            }
        ],
        "lsid" : {
            "id" : UUID("07b43083-8ab9-4bcb-8768-919a3f27655f")
        },
        "$clusterTime" : {
            "clusterTime" : Timestamp(1598875647, 409),
            "signature" : {
                "hash" : BinData(0,"+/YcdPyQriT8RL1LtFUhxe2BtCE="),
                "keyId" : NumberLong("6861636045532823556")
            }
        },
        "$db" : "test"
    },
    "msg" : "Index Build: draining writes received during build",
    "numYields" : 0,
    "locks" : {

    },
    "waitingForLock" : false,
    "lockStats" : {
        "ReplicationStateTransition" : {
            "acquireCount" : {
                "w" : NumberLong(6)
            }
        },
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(4)
            }
        },
        "Database" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(4)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(3),
                "W" : NumberLong(1)
            }
        },
        "Mutex" : {
            "acquireCount" : {
                "r" : NumberLong(4)
            }
        }
    },
    "waitingForFlowControl" : false,
    "flowControlStats" : {
        "acquireCount" : NumberLong(3),
        "timeAcquiringMicros" : NumberLong(1)
    }
}

This Index Build: draining writes received during build makes no sense since there was no read/writes to the test collection during index creation.

Also index creation hangs only in non-empty collection. Index creates successfully in empty collection.

What might be an issue in this case? I'm out of ideas.


Solution

  • Finally figured it out with the help of MongoDB team. The node can't communicate with itself so it will hang trying to commit the index build. This was the reason. Adding keyfile fixed the issue:

    rm -f mongo.keyfile
    openssl rand -base64 756 > mongo.keyfile
    chmod 400 mongo.keyfile
     
    bin/mongod --config mongo.conf --keyFile mongo.keyfile
    

    Here are the links to the MongoDB's Jira issues which cover this subject: https://jira.mongodb.org/browse/SERVER-50665 and https://jira.mongodb.org/browse/SERVER-48516