We have a MarkLogic 9.x server with 4.1.x Data Hub we want to practice disaster recovery. This activity assumes the server is completely unrecoverable and we need to stand up an all new server with a clean install of ML Server & Data Hub and then restore all the databases from backups. Replicating databases to another server in production is not an option.
Looking through the ML doco & knowledgebase, I was unable to find how this would be done.
Can it be done, and if so is there also a specific sequence of database restores required or any databases that don't need to be restored? Assume all databases on the server have viable full backups before the server was lost.
Restoring a backup is rather straightforward, assuming that you have the same topology and forest naming convention in the disaster recovery cluster.
The procedures for initiating a backup and restoring are described in the Admin Guide: Backing Up and Restoring a Database
You didn't mention whether these would be full backups, incrementals, or if journal archives were enabled and available, but they are discussed in the Concepts Guide: High Availability and Disaster Recovery
MarkLogic supports online backups and restores, so you can protect and restore your data without bringing the system offline or halting queries or updates. Backups are initiated via administration calls, either via the web console or an XQuery script. You specify a database to backup and a target location. Backing up a database backs up its configuration files, all the forests in the database, as well as the corresponding security and schemas databases. It's particularly important to backup the security database because MarkLogic tracks role identifiers as xs:long values and the backup forest data can't be read without the corresponding roles existing in the security database.
You can also choose to selectively backup an individual forest instead of an entire database. That's a convenient option if only the data in one forest is changing.
The topics in this section are as follows:
You will want to restore your content databases (and maybe Modules, but should have the code available and able to deploy), but it is not recommended to restore some of the build-in databases such as app-services: https://help.marklogic.com/Knowledgebase/Article/View/why-shouldnt-you-restore-the-app-services-database-from-another-cluster
Best Practice: The "App-Services" database is one of the default MarkLogic databases, used to track configuration file timestamp information local to the cluster on which it resides. It is not recommended to restore "App-Services" databases across MarkLogic clusters.
and extra precautions about Security DB: https://help.marklogic.com/Knowledgebase/Article/View/restoring-security-database
Note: It is recommended that a backup of the security database from the new cluster is created and saved before performing the restore of a Security database from a different cluster.