Most guidelines recommend to use mongodump/mongorestore, but for large product databases downtime can be very long
You can use replication and an additional server for this or the same server if the load allows.
You need 3 running MongoDB instance:
Anyway you need to backup your database. You can run “mongodump” without parameters and directory “./dump” will be created with the database dump. You can use “--gzip“ parameter to compress result size.
mongodump --gzip
Just in case, the command to restore:
mongorestore --gzip
It should be run in the same directory where “./dump” dir and “--gzip“ parameter should be added if used in “mongodump”.
Begin configure from the additional server. My target system is Linux RedHat without Internet, so I download and install MongoDB via RPM manually. Add the section to /etc/mongod.conf:
replication:
oplogSizeMB: 10240
replSetName: REPLICA
Check that the net section look like this to allow access from other servers:
net:
bindIp: 0.0.0.0
port: 27017
and run:
service mongod start
Run the third MongoDB instance - arbiter. It can work on the additional server on a different port. Create a temporary directory for the arbiter database:
mkdir /tmp/mongo
chmod 777 -R /tmp/mongo
and run:
mongod --dbpath /tmp/mongo --port 27001 --replSet REPLICA \
--fork --logpath /tmp/mongo/db1.log
Now configure the main server. Edit /etc/mongod.conf
replication:
oplogSizeMB: 10240
replSetName: REPLICA
and restart MongoDB on the main server:
service mongod restart
It’s important! After restarting the main server read operations may be unavailable. I was getting the following error:
{ "ok" : 0, "errmsg" : "node is recovering", "code" : 13436 }
So as quickly as possible you need to connect to MongoDB on the main server via “mongo” console and run the following command to configure replication:
rs.initiate(
{
_id: "REPLICA",
members: [
{ _id: 0, host : "<IP address of main server>:27017",
priority: 1.0 },
{ _id: 1, host : "<IP address of additional server>:27017",
priority: 0.5 },
{ _id: 2, host : "<IP address of additional server(the arbiter)>:27001",
arbiterOnly : true, priority: 0.5 }
]
}
)
After this operation all actions with MongoDB will be available and data synchronization will be started.
I don’t recommend to use rs.initiate() on the main server without parameters as in most tutorials, because name of the main server will be configured by default as DNS-name from the /etc/hostname. It's not very convenient for me because I use IP-addresses for communications in my projects.
To check the synchronization progress you can call from “mongo” console:
rs.status()
Result example:
{
"set" : "REPLICA",
"date" : ISODate("2017-01-19T14:30:34.292Z"),
"myState" : 1,
"term" : NumberLong(1),
"heartbeatIntervalMillis" : NumberLong(2000),
"members" : [
{
"_id" : 0,
"name" : "<IP address of main server>:27017",
"health" : 1.0,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 165,
"optime" : {
"ts" : Timestamp(6377323060650835, 3),
"t" : NumberLong(1)
},
"optimeDate" : ISODate("2017-01-19T14:30:33.000Z"),
"infoMessage" : "could not find member to sync from",
"electionTime" : Timestamp(6377322974751490, 1),
"electionDate" : ISODate("2017-01-19T14:30:13.000Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "<IP address of additional server>:27017",
"health" : 1.0,
"state" : 5,
"stateStr" : "STARTUP2",
"uptime" : 30,
"optime" : {
"ts" : Timestamp(0, 0),
"t" : NumberLong(-1)
},
"optimeDate" : ISODate("1970-01-01T00:00:00.000Z"),
"lastHeartbeat" : ISODate("2017-01-19T14:30:33.892Z"),
"lastHeartbeatRecv" : ISODate("2017-01-19T14:30:34.168Z"),
"pingMs" : NumberLong(3),
"syncingTo" : "<IP address of main server>:27017",
"configVersion" : 1
},
{
"_id" : 2,
"name" : "<IP address of additional server (the arbiter)>:27001",
"health" : 1.0,
"state" : 7,
"stateStr" : "ARBITER",
"uptime" : 30,
"lastHeartbeat" : ISODate("2017-01-19T14:30:33.841Z"),
"lastHeartbeatRecv" : ISODate("2017-01-19T14:30:30.158Z"),
"pingMs" : NumberLong(0),
"configVersion" : 1
}
],
"ok" : 1.0
}
After “stateStr” of the additional server will be replaced from ”STARTUP2” to ”SECONDARY”, our servers are synchronized.
While we wait for the end of the synchronization, it is necessary to modify client applications a little bit they can work with all servers in replica.
If you use the ConnectionString, you should replace it with something like:
mongodb://<IP address of main server>:27017,<IP address of additional server>:27017,<IP address of additional server (the arbiter)>:27001/?replicaSet=REPLICA
If you use C++ mongo-cxx-driver legacy, as I am, you should to use mongo::DBClientReplicaSet instead mongo::DBClientConnection and list all three servers in connection parameters, including the arbiter.
There is a third option - you can simply change IP of MongoDB server in clients after switching PRIMARY-SECONDARY, but it's not very fair.
After the synchronization has ended and an additional server status has established as SECONDARY, we need to switch the PRIMARY and SECONDARY by executing the command in “mongo” console on the main server. This is important because command will not work on the additional server.
cfg = rs.conf()
cfg.members[0].priority = 0.5
cfg.members[1].priority = 1
cfg.members[2].priority = 0.5
rs.reconfig(cfg)
Then check server status by executing:
rs.status()
Stop the MongoDB on the main server
service mongod stop
and simply delete the entire contents of a directory with database. It is safe, because we have a working copy on the additional server, and in the beginning we have made a backup. Be careful. MongoDB doesn’t create a database directory itself. If you've deleted it, you need not only to restore
mkdir /var/lib/mongo
and setup owner:
chown -R mongod:mongod /var/lib/mongo
Check storage engine wiredTiger is configured in /etc/mongod.conf. From 3.2 it is used by default:
storage:
...
engine: wiredTiger
...
And run MongoDB:
service mongod start
The main server will get the configuration from the secondary server automatically and data will be synced back to WiredTiger storage.
After the synchronization is finished switch the PRIMARY server back. This operation should be performed on an additional server because it is the PRIMARY now.
cfg = rs.conf()
cfg.members[0].priority = 1
cfg.members[1].priority = 0.5
cfg.members[2].priority = 0.5
rs.reconfig(cfg)
Return the old version of database clients or change ConnectionString back.
Now turn off replication if necessary. Remove 2 replication servers from the main server:
rs.remove("<IP address of additional server>:27017")
rs.remove("<IP address of additional server (the arbiter)>:27001")
Remove all “replication” section from /etc/mongod.conf and restart MongoDB:
service mongod restart
After these we get the warning when connected via the “mongo” console:
2017-01-19T12:26:51.948+0300 I STORAGE [initandlisten] ** WARNING: mongod started without --replSet yet 1 documents are present in local.system.replset
2017-01-19T12:26:51.948+0300 I STORAGE [initandlisten] ** Restart with --replSet unless you are doing maintenance and no other clients are connected.
2017-01-19T12:26:51.948+0300 I STORAGE [initandlisten] ** The TTL collection monitor will not start because of this.
To get rid of it, you need to remove the database “local”. There is only one collection “startup_log” in this database in default state, so you can do this without fear via “mongo” console
use local
db.dropDatabase()
and restart MongoDB:
service mongod restart
If you will remove the “local” database before “replication” section from /etc/mongod.conf, it is immediately restored. So I could not do only one MongoDB restart.
On the additional server perform the same action:
The arbiter just stop and remove:
pkill -f /tmp/mongo
rm -r /tmp/mongo