ApsaraDB for MongoDB:Restore data of an ApsaraDB for MongoDB instance to a self-managed MongoDB database by using physical backup

Prerequisites

• The ApsaraDB for MongoDB instance is a replica set instance.
• The instance uses local disks to store data.
• The Transparent Data Encryption (TDE) feature is disabled for the ApsaraDB for MongoDB instance. For more information, see Configure TDE for an ApsaraDB for MongoDB instance.
• The storage engine of the ApsaraDB for MongoDB instance is WiredTiger or RocksDB. If the storage engine is TerarkDB, use logical backup to restore data of the ApsaraDB for MongoDB instance to a self-managed MongoDB database. For more information, see Restore data of an ApsaraDB for MongoDB instance to a self-managed MongoDB database by using logical backup.
  Note

  • You can view the storage engine of an ApsaraDB for MongoDB instance on the Basic Information page in the ApsaraDB for MongoDB console.
  • If the storage engine is RocksDB, you must compile and install a MongoDB application that is equipped with the RocksDB storage engine.

Database version requirements

The version of the ApsaraDB for MongoDB instance must correspond to the version of the self-managed MongoDB database. The following table lists the mappings between the ApsaraDB for MongoDB instance and the self-managed MongoDB database.

Formats of physical backup files

Environment

The following procedure uses an ECS instance created from an Ubuntu 16.04 64-bit image. For more information, see Create an ECS instance.

Step 1: Configure environment variables

Configure environment variables for the self-managed MongoDB database. This way, you do not need to enter executable file paths when you run commands. Before you perform this step, make sure that MongoDB is installed. For more information, visit Install MongoDB.

1. Run the following command to open the profile file in Linux:

   sudo vi /etc/profile

2. Press the I key to enter the edit mode. Then, enter the following code in the last line:

   export PATH=$PATH:/<The path of the MongoDB server>/bin

   Example:

   export PATH=$PATH:/test/mongo/bin

   Note /test/mongo/bin is used as the path of the MongoDB server in this example. Use the actual path when you configure environment variables.
3. Press the Esc key to exit the edit mode, and enter :wq to save the file and exit.
4. Run the following command to make the configured environment variables take effect:

   source /etc/profile

Step 2: Download and decompress a physical backup file

1. Download the physical backup data of the ApsaraDB for MongoDB instance in the ApsaraDB for MongoDB console. For more information, see Download the physical backup data of a replica set instance. You can also run the following command to download the data:

   wget -c '<External download URL of the data backup file>' -O <Custom name of the downloaded data backup file>.<File name extension>

   Note Make sure that the file extension is .tar.gz or _qp.xb.
2. Run the following command to create a directory named data in the /test/mongo/ directory. Then, move the downloaded physical backup file of the ApsaraDB for MongoDB instance to the /test/mongo/data/ directory.

   mkdir -p /test/mongo/data && mv <Name of the physical backup file.Extension> /test/mongo/data

3. Decompress the physical backup file.
   • If the physical backup file has a .tar.gz extension, such as hins20190412.tar.gz, run the following command to decompress the file:

     cd /test/mongo/data/ && tar xzvf hins20190412.tar.gz 

   • If the physical backup file has a _qp.xb extension, such as hins20190412_qp.xb, perform the following operations to decompress the file:
     1. Install the Percona XtraBackup and qpress tools. For more information, visit Installing Percona XtraBackup on Debian and Ubuntu.
     2. Decompress the physical backup file. In this example, the file is hins20190412_qp.xb.

        # Go to the directory where the file is located. 
        cd /test/mongo/data/
        # Decompress the package. 
        cat hins20190412_qp.xb | xbstream -x -v
        # Decompress the physical backup file. 
        innobackupex --decompress --remove-original /test/mongo/data

Step 3: Restore data to the self-managed MongoDB database in standalone mode

1. Run the following command to create a configuration file named mongod.conf in the /test/mongo directory:

   touch /test/mongo/mongod.conf

2. On the command line, enter vi /test/mongo/mongod.conf to open the mongod.conf file. Press the I key to enter the edit mode.

   You can select a configuration template based on the storage engine of the ApsaraDB for MongoDB instance, and copy the selected configuration template to the mongod.conf configuration file.

   Note In this file, enable standalone startup and authorization.
   • WiredTiger

     systemLog:
         destination: file
         path: /test/mongo/mongod.log
         logAppend: true
     security:
         authorization: enabled
     storage:
         dbPath: /test/mongo/data
         directoryPerDB: true
     net:
         port: 27017
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod.pid

     Note By default, ApsaraDB for MongoDB uses the WiredTiger storage engine, and the directoryPerDB option is enabled. Therefore, the directoryPerDB option is set to true in the preceding configuration.
   • RocksDB

     systemLog:
         destination: file
         path: /test/mongo/logs/mongod.log
         logAppend: true
     security:
         authorization: enabled
     storage:
         dbPath: /test/mongo/data
             engine: rocksdb
     net:
         port: 27017
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod.pid

3. Press the Esc key to exit the edit mode, and enter :wq to save the file and exit.
4. Use the mongod.conf configuration file to start MongoDB.

   mongod -f /test/mongo/mongod.conf

5. After you start MongoDB, run the following command to log on to the self-managed MongoDB database and go to the mongo shell:

   mongo --host 127.0.0.1 -u <username> -p <password> --authenticationDatabase admin

   Note:

   • <username>: the account used to log on to the self-managed MongoDB database. The default value is root.
   • <password>: the password used to log on to the self-managed MongoDB database.
     Note If the password contains special characters, you must enclose the password in single quotation marks (' '). Example: 'test123!@#' Otherwise, you may fail to log on to the database.
6. Run the show dbs command in the mongo shell to query all the available databases on the MongoDB server. This way, you can check whether the restoration is successful.
7. Run the exit command in the mongo shell to exit the mongo shell.

Step 4: Start the self-managed MongoDB database in replica set mode

By default, the physical backup file of the ApsaraDB for MongoDB instance contains the replica set configuration. You must start MongoDB in standalone mode. Otherwise, the self-managed MongoDB database may be inaccessible.

If you want to start MongoDB in replica set mode, you must restore data to the self-managed MongoDB database in standalone mode before you perform the following operations:

1. On the command line, log on to the database as the test user by using the mongo shell:

   mongo --host 127.0.0.1 -u test -p <Password of the test account> --authenticationDatabase admin

   Note If the password contains special characters, you must enclose the password in single quotation marks (' '). Example: 'test123!@#' Otherwise, you may fail to log on to the database.
2. After you log on to the database, run the commands in the sample code to perform the following operations:
   1. Create a temporary user in the admin database and grant the temporary user the read and write permissions on the local database.
   2. Switch to the temporary user and delete the original replica set configuration of the local database.
   3. Switch back to the test user and delete the temporary user and the temporary permissions.
      Note Replace <Password of the test account> in the following code with the password of your test account.

   use admin
   db.runCommand({
       createRole: "tmprole",
       roles: [
           {
               role: "test",
               db: "admin"
           }
       ],
       privileges: [
           {
               resource: {
                   db: 'local',
                   collection: 'system.replset'
               },
               actions: [
                   'remove'
               ]
           }
       ]
   })
   db.runCommand({
       createUser: "tmpuser",
       pwd: "tmppwd",
       roles: [
           'tmprole'
       ]
   })
   db.auth('tmpuser','tmppwd')
   use local
   db.system.replset.remove({})
   use admin
   db.auth('test','<Password of the test account>')
   db.dropRole('tmprole')
   db.dropUser('tmpuser')

3. Run the following commands to shut down the MongoDB service and exit the mongo shell:

   use admin
   db.shutdownServer()
   exit

4. Create the authentication file on the replica set.

   To start MongoDB in replica set mode, you must create a key file for all replica set nodes to authenticate each other.

   1. Run the following command to create the keyFile folder in the mongo directory as the authentication file directory and create a key file in the authentication file directory.

      mkdir -p /test/mongo/keyFile && touch /test/mongo/keyFile/mongodb.key

   2. Run the vi /test/mongo/keyFile/mongodb.key command to open the mongodb.key file. Press the I key to enter the edit mode. Example:

      MongoDB Encrypting File

      Note The key that is used to encrypt data has the following limits:

      • The key must be 6 to 1,024 characters in length.
      • The key can contain only Base64-encoded characters.
      • The key cannot contain equal signs (=).
   3. Press the Esc key to exit the edit mode, and enter :wq to save the file and exit.
   4. On the command line, run the following command to change the permission on the authentication file to 400. This way, only the owner of the file can view the content of the file.

      sudo chmod 400 /test/mongo/keyFile/mongodb.key

   Note This authentication file applies to all replica set nodes.
5. Perform the following operations to prepare two empty nodes for the replica set:
   1. Run the following command to create two copies of the mongod.conf file. The two copies are used as the configuration files for the other two nodes.

      cp /test/mongo/mongod.conf /test/mongo/mongod1.conf && cp /test/mongo/mongod.conf /test/mongo/mongod2.conf

   2. Run the following command to create data directories for the other two nodes:

      mkdir -p /test/mongo/data1 && mkdir -p /test/mongo/data2

6. Modify the configuration file of each node by performing the following operations:
   • Run the vi /test/mongo/mongod.conf command to open the configuration file of Node 1. Modify the file based on the following content. Then, save and exit the configuration file.

     systemLog:
         destination: file
         path: /test/mongo/mongod.log
         logAppend: true
     security:
         authorization: enabled
         keyFile: /test/mongo/keyFile/mongodb.key
     storage:
         dbPath: /test/mongo/data
         directoryPerDB: true
     net:
         bindIp: 127.0.0.1
         port: 27017
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod.pid
     replication:
         replSetName: "rs0"

   • Run the vi /test/mongo/mongod1.conf command to open the configuration file of Node 2. Modify the file based on the following content. Then, save and exit the configuration file.

     systemLog:
         destination: file
         path: /test/mongo/mongod1.log
         logAppend: true
     security:
         authorization: enabled
         keyFile: /test/mongo/keyFile/mongodb.key
     storage:
         dbPath: /test/mongo/data1
         directoryPerDB: true
     net:
         bindIp: 127.0.0.1
         port: 27018
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod1.pid
     replication:
         replSetName: "rs0"

   • Run the vi /test/mongo/mongod2.conf command to open the configuration file of Node 3. Modify the file based on the following content. Then, save and exit the configuration file.

     systemLog:
         destination: file
         path: /test/mongo/mongod2.log
         logAppend: true
     security:
         authorization: enabled
         keyFile: /test/mongo/keyFile/mongodb.key
     storage:
         dbPath: /test/mongo/data2
         directoryPerDB: true
     net:
         bindIp: 127.0.0.1
         port: 27019
         unixDomainSocket:
             enabled: false
     processManagement:
         fork: true
         pidFilePath: /test/mongo/mongod2.pid
     replication:
         replSetName: "rs0"

   Note Parameter description:

   • path in systemLog.path: the path to store the MongoDB log files in the current node.
   • dbpath: the path to store the MongoDB data files in the current node.
   • pidFilePath: the path to store the MongoDB process ID (PID) files in the current node.
   • keyFile: the path to store the authentication file for the replica set. All nodes in the replica set must use the same authentication file.
   • bindIp: the IP address of the current node. If all nodes in the replica set are deployed on the same server, they can use the same IP address.
   • port: the port number of the current node. If all nodes in the replica set are deployed on the same server, they must use different port numbers.
   • replication: the configuration of the replica set.
   • replSetName: the name of the replica set.
7. Run the following command to start the three nodes:

   mongod -f /test/mongo/mongod.conf && mongod -f /test/mongo/mongod1.conf && mongod -f /test/mongo/mongod2.conf

8. After you start the three nodes, use the test account to log on to the self-managed MongoDB database.

   mongo --host 127.0.0.1 -u test -p <Password of the test account> --authenticationDatabase admin

   Note If the password contains special characters, you must enclose the password in single quotation marks (' '). Example: 'test123!@#' Otherwise, you may fail to log on to the database.
9. In the mongo shell, add the nodes that you created in the preceding operations to the replica set and initialize the replica set.

   rs.initiate( {
      _id : "rs0",
      version : 1,
      members: [
         { _id: 0, host: "127.0.0.1:27017" , priority : 1},
         { _id: 1, host: "127.0.0.1:27018" , priority : 0},
         { _id: 2, host: "127.0.0.1:27019" , priority : 0}
      ]
   })

   Example of successful initialization:
   Note The rs.initiate() command is used in this step. For more information about the command, visit rs.initiate().

   After the command is executed, data is synchronized between the two added nodes and the primary node. The time consumed in this process varies based on the size of the backup file. After the data is synchronized, the self-managed MongoDB database is started in replica set mode.

10. Perform the following operations to check whether the self-managed MongoDB database in replica set mode is started:
    1. Run the exit command to exit the mongo shell.
    2. Run the following command to log on to the self-managed MongoDB database again:

       mongo -u <username> -p <password> --authenticationDatabase admin

       Note:

       • <username>: the account used to log on to the self-managed MongoDB database. The default value is root.
       • <password>: the password used to log on to the self-managed MongoDB database.
         Note If the password contains special characters, you must enclose the password in single quotation marks (' '). Example: 'test123!@#' Otherwise, you may fail to log on to the database.
    3. Check whether the self-managed MongoDB database is started in replica set mode. If <Name of the replica set>:PRIMARY> is displayed on the left side of the command line in the mongo shell, the self-managed MongoDB database is started in replica set mode.

FAQ

Why does an error occur when I use the specified mongod.conf configuration file to start the self-managed MongoDB database?

Why does an error occur when I use the mongod.conf configuration file to start the self-managed MongoDB database in replica set mode?

You may fail to change the permission on the specified keyFile authentication file to 600. On the command line, run the sudo chmod 600 <Path of keyFile> command to modify the permissions.

Why does the system performance become low after I start the self-managed MongoDB database in replica set mode?

After you start MongoDB in replica set mode, the system automatically starts to synchronize data of the primary node to other nodes. This process can affect the system performance. The system performance returns to normal after the data is synchronized.