Background
Sharded mongo cluster with 5 shards
3 stand alone config server instances
Mongo version : 3.0.8
Requirement
Upgrading to 3.2.8 as well as converting individual config servers to replica set
Reference
Config servers to replica set conversion
Preparation
Tested successfully in QA environment.
Problem
The production scale is way more than QA and it is serving live traffic. Is there any way to have a roll back plan if the conversion from individual servers to replica set fails.
One possible solution can be : To back up the config server data and restore in case of failure, then restart 3 servers and mongos as well. How feasible is this ?
The best and most secure option was to back up entire cluster. This can be done manually or using cloud manager. Using cloud manager a backup schedule can be configured as well. The size of DB was huge so we did not wait for entire cluster to back up and went ahead with backing up config servers only (NOT RECOMMENDED AT ALL). Fortunately we did not encounter any problems and everything went pretty smooth. Thanks to cloud manager.