Migrate ApsaraDB for MongoDB to Lindorm

Prerequisites

• Migration tasks are currently supported only within the Germany (Frankfurt) region.

• ApsaraDB for MongoDB utilizes either ReplicaSet or sharded cluster architecture.

  Important

  If the source database employs a sharded cluster architecture with ApsaraDB for MongoDB, you must obtain connection addresses for all shard nodes, ensuring that the accounts and passwords for each shard are identical. For guidance on how to apply, refer to Requesting Shard or ConfigServer Node Connection Addresses.

• A Lindorm instance with a wide table engine has been successfully created. For methods on how to create an instance, see Create an instance.

• Wide tables have been established in Lindorm to meet business requirements. For instructions on creation, see how to connect and use the wide table engine via Lindorm-cli or how to access the wide table engine using Lindorm Shell.

  Note

  When creating a wide table in Lindorm using HBase, it is advisable to establish column mapping relationships (where the wide table columns are standard columns). For more information, see how to add column mapping to an HBase table.

• The storage space of the target instance should be at least 10% larger than the source database's used storage space.

Notes

Billing overview

Migration types

Permissions required for database accounts

Procedure

This operation uses a wide table created by Lindorm SQL as an example.

1. Use one of the following methods to go to the Data Migration page and select the region in which the data migration instance resides.

   DTS console

   DMS console

   1. Log on to the DTS console.

   2. In the left-side navigation pane, click Data Migration.

   3. In the upper-left corner of the page, select the region in which the data migration instance resides.

   Note

   The actual operation may vary based on the mode and layout of the DMS console. For more information, see Simple mode and Customize the layout and style of the DMS console.

   1. Log on to the DMS console.

   2. In the top navigation bar, move the pointer over Data Development > DTS (DTS) > Data Migration .

   3. From the drop-down list to the right of Data Migration Tasks, select the region in which the data synchronization instance resides.

2. Click Create Task to go to the task configuration page.

3. Optional. Click New Configuration Page in the upper-right corner of the page.

   Note

   1. Skip this step if the Back to Previous Version button is displayed in the upper-right corner of the page.

   2. Specific parameters in the new and previous versions of the configuration page may be different. We recommend that you use the new version of the configuration page.

4. Configure the source and destination databases. The following table describes the parameters.

   Category	Configuration	Description

   Category	Configuration	Description
   None	Task Name	The name of the DTS task. DTS automatically generates a task name. We recommend that you specify a descriptive name that makes it easy to identify the task. You do not need to specify a unique task name.
   Source Database	Select Existing Connection	The instance that you want to use. You can choose whether to use an existing instance based on your business requirements. If you select an existing instance, DTS automatically populates the parameters for the database. If you do not use an existing instance, you must configure the database information below. Note You can register a database with DTS on the Database Connections page or the new configuration page. For more information, see Manage database connections. In the DMS console, you can select an existing database from the Select a DMS database instance. drop-down list. You can also click Add DMS Database Instance or go back to the homepage of the DMS console to register a database with DMS. For more information, see Register an Alibaba Cloud database instance and Register a database hosted on a third-party cloud service or a self-managed database.
   	Database Type	Select MongoDB.
   	Access Method	Select Cloud Instance.
   	Instance Region	Select the region where the source ApsaraDB for MongoDB resides.
   	Replicate Data Across Alibaba Cloud Accounts	In this example, the migration is performed within the same Alibaba Cloud account. Select No.
   	Architecture	In this example, select Replicaset Architecture. Note If Architecture is selected as Sharded Cluster, you also need to enter the Shard account and Shard password.
   	Migration Method	Select the method for incremental data migration based on your actual situation. Oplog (recommended): If the source database has enabled Oplog, this option is supported. Note Local self-managed MongoDB and ApsaraDB for MongoDB have Oplog enabled by default. When migrating incremental data using this method, the delay of the incremental migration task is small (the speed of pulling logs is fast). Therefore, it is recommended to select Oplog. ChangeStream: If the source database has enabled Change Streams (Change Streams), this option is supported. Note When the source database is Amazon DocumentDB (non-elastic cluster), only ChangeStream is supported. When the Architecture of the source database is selected as Sharded Cluster, there is no need to fill in the Shard account and Shard password.
   	Instance ID	Select the instance ID of the source ApsaraDB for MongoDB.
   	Authentication Database Name	Enter the name of the database to which the database account of the source ApsaraDB for MongoDB instance belongs. If it has not been modified, the default value is admin.
   	Database Account	Enter the database account of the source ApsaraDB for MongoDB.
   	Database Password	The password that is used to access the database.
   	Encryption	Specifies whether to encrypt the connection to the source database. You can select Non-encrypted, SSL-encrypted, or Mongo Atlas SSL based on your business requirements. The options available for the Encryption parameter are determined by the values selected for the Access Method and Architecture parameters. The options displayed in the DTS console prevail. Note If the Architecture parameter is set to Sharded Cluster, and the Migration Method parameter is set to Oplog for the ApsaraDB for MongoDB database, the Encryption parameter SSL-encrypted is unavailable. If the source database is a self-managed MongoDB database that uses the Replica Set architecture, the Access Method parameter is not set to Alibaba Cloud Instance, and the Encryption parameter is set to SSL-encrypted, you can upload a certification authority (CA) certificate to verify the connection to the source database.
   Destination Database	Select Existing Connection	The instance that you want to use. You can choose whether to use an existing instance based on your business requirements. If you select an existing instance, DTS automatically populates the parameters for the database. If you do not use an existing instance, you must configure the database information below. Note You can register a database with DTS on the Database Connections page or the new configuration page. For more information, see Manage database connections. In the DMS console, you can select an existing database from the Select a DMS database instance. drop-down list. You can also click Add DMS Database Instance or go back to the homepage of the DMS console to register a database with DMS. For more information, see Register an Alibaba Cloud database instance and Register a database hosted on a third-party cloud service or a self-managed database.
   	Database Type	Select Lindorm.
   	Access Method	Select Cloud Instance.
   	Instance Region	Select the region where the target Lindorm resides.
   	Instance ID	Select the instance ID of the target Lindorm.
   	Database Account	Enter the database account of the target Lindorm.
   	Database Password	The password that is used to access the database.

5. Once the configuration is complete, click the Test Connectivity and Proceed button at the bottom of the page.

   Note

   • Ensure that the DTS service's IP address range is added, either automatically or manually, to the security settings of both source and target databases to permit access by the DTS server. For more information, see how to add the IP address range of the DTS server.

   • If either the source or target database is self-managed (the Access Method is not Alibaba Cloud Instance), you must also click Test Connectivity in the CIDR Blocks of DTS Servers dialog box that appears.

6. Configure the objects to be migrated.

   1. On the Configure Objects page, configure the objects that you want to migrate.

      Configuration	Description

      Configuration	Description
      Migration Types	To perform only full data migration, select only Full Data Migration. To ensure service continuity during data migration, select Full Data Migration and Incremental Data Migration. Note If you do not select Incremental Data Migration, we recommend that you do not write data to the source database during data migration. This ensures data consistency between the source and destination databases.
      Processing Mode of Conflicting Tables	No configuration needed. Keep the default setting.
      Capitalization of Object Names in Destination Instance	The capitalization of database names and collection names in the destination instance. By default, DTS default policy is selected. You can select another option to ensure that the capitalization of object names is the same as that of the source or destination database. For more information, see Specify the capitalization of object names in the destination instance.
      Source Objects	In the Source Database Objects box, select the collections you want to migrate, and then click to transfer them to the Selected Objects box.
      Selected Objects	If the wide table in the target database is created through Lindorm SQL, you need to migrate data by configuring newly added columns. Columns that are not configured will not be migrated to the target database. Edit schema mapping. Right-click the database containing the collection you want to migrate in Selected Objects. Change the Schema Name to match the target database's name in Lindorm. Optional: In the Select DDL and DML Operations to Be Synchronized section, choose the operations you want to migrate incrementally. Click OK . Edit table name mapping. Right-click the collection you want to migrate in Selected Objects. Change the Table Name to the name of the desired table in Lindorm. Optional: Define filter conditions. For more information, see how to set filter conditions. Optional: In the Select DDL and DML Operations to Be Synchronized area, you can select the operations you want to incrementally migrate. Configure the fields to be migrated from MongoDB. By default, DTS automatically maps the data from the collections designated for migration. You should verify that the mapping expressions in the Assignment column align with your requirements and configure the Column Name , Type , Length , and Precision accordingly. In the Assignment column, you can view the field name corresponding to the row data in MongoDB using the bson_value() expression. The "" represents a field name in MongoDB. For instance, when the expression is bson_value("age"), it refers to the row data corresponding to the age field within MongoDB. Optional: Delete fields that do not need to be migrated. Note For fields that do not need to be migrated, you can click after the row data. Configure the fields to be migrated. You can proceed with subsequent operations if the bson_value() expression meets your requirements. Fields that meet the requirements of the expression Fields that do not meet the requirements of the expression Enter the Column Name. Note Enter the column name of the target table in Lindorm. If you created the target table using SQL, enter its column name in Lindorm in the Column Name field. If you're working with a target table created in HBase and want to utilize the newly introduced column feature, it's necessary to establish column mapping relationships beforehand. For more information, see Example of adding column mapping to an HBase table. Input the following in the Column Name: If the column functions as a primary key, enter ROW . If the column is not a primary key, use the format Column Family:Column Name, such as person:name. Select the Type of the column data. Important Make sure that the data type of the target table is compatible with the data in the source MongoDB. Optional: You can configure the Length and Precision settings for the column data. Repeat the preceding operations to map the relevant fields one by one. Note For example, fields with hierarchical relationships (parent-child structures). In the Operation column, click the icon following the row data. Click the + Add Column button. Configure the Column Name, Type, Length, and Precision. In the text box under Assignment, enter the bson_value() expression. For more information, see the example of assignment configuration. Important The target table's primary key column must be assigned the value bson_value("_id"). When setting up the bson_value() expression, ensure you configure it down to the smallest subfield based on the hierarchical structure to prevent data loss or task failure. Repeat the preceding operations to map the relevant fields one by one. Click OK .

   2. Click Next: Advanced Settings to configure advanced settings.

      Configuration	Description

      Configuration	Description
      Dedicated Cluster for Task Scheduling	By default, DTS schedules the data migration task to the shared cluster if you do not specify a dedicated cluster. If you want to improve the stability of data migration tasks, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster.
      Retry Time for Failed Connections	The retry time range for failed connections. If the source or destination database fails to be connected after the data migration task is started, DTS immediately retries a connection within the retry time range. Valid values: 10 to 1440. Unit: minutes. Default value: 720. We recommend that you set the parameter to a value greater than 30. If DTS is reconnected to the source and destination databases within the specified retry time range, DTS resumes the data migration task. Otherwise, the data migration task fails. Note If you specify different retry time ranges for multiple data migration tasks that share the same source or destination database, the value that is specified later takes precedence. When DTS retries a connection, you are charged for the DTS instance. We recommend that you specify the retry time range based on your business requirements. You can also release the DTS instance at the earliest opportunity after the source database and destination instance are released.
      Retry Time for Other Issues	The retry time range for other issues. For example, if DDL or DML operations fail to be performed after the data migration task is started, DTS immediately retries the operations within the retry time range. Valid values: 1 to 1440. Unit: minutes. Default value: 10. We recommend that you set the parameter to a value greater than 10. If the failed operations are successfully performed within the specified retry time range, DTS resumes the data migration task. Otherwise, the data migration task fails. Important The value of the Retry Time for Other Issues parameter must be smaller than the value of the Retry Time for Failed Connections parameter.
      Enable Throttling for Full Data Migration	Specifies whether to enable throttling for full data migration. During full data migration, DTS uses the read and write resources of the source and destination databases. This may increase the loads of the database servers. You can enable throttling for full data migration based on your business requirements. To configure throttling, you must configure the Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) parameters. This reduces the loads of the destination database server. Note You can configure this parameter only if you select Full Data Migration for the Migration Types parameter.
      Only one data type for primary key _id in a single table	For the data to be migrated, whether the data type of the primary key _id is unique within the same collection. Note This configuration is only available when the Migration Types is set to Full Data Migration. Yes: Unique. DTS does not scan the data type of the primary key in the source data during the full migration phase from the source database. No: Indicates non-uniqueness. During the full migration phase, DTS scans the data type of the primary key in the source data slated for migration from the source database.
      Enable Throttling for Incremental Data Migration	Specifies whether to enable throttling for incremental data migration. To configure throttling, you must configure the RPS of Incremental Data Migration and Data migration speed for incremental migration (MB/s) parameters. This reduces the loads of the destination database server. Note You can configure this parameter only if you select Incremental Data Migration for the Migration Types parameter.
      Environment Tag	Select an environment tag to identify the instance based on your needs. In this example, no environment tag is selected.
      Configure ETL	Specifies whether to enable the extract, transform, and load (ETL) feature. For more information, see What is ETL? Valid values: Yes: configures the ETL feature. You can enter data processing statements in the code editor. For more information, see Configure ETL in a data migration or data synchronization task. No: does not configure the ETL feature. Note If the target table is created using HBase, consider the following: The ETL syntax specifies which columns to include and which to exclude. During migration, MongoDB document fields that are configured for ETL are stored in the default HBase column family 'f'. For instance, all top-level elements, except for _id and name, are dynamically written to the destination table. For more information, see the HBase table migration example (ETL). script:e_expand_bson_value("*", "_id,name") If using both newly added columns and ETL, ensure there is no data duplication in Lindorm. Columns not configured with newly added columns or ETL will not be migrated to the target database.
      Monitoring and Alerting	Specifies whether to configure alerting for the data migration task. If the task fails or the migration latency exceeds the specified threshold, the alert contacts receive notifications. Valid values: No: does not configure alerting. Yes: configures alerting. In this case, you must also configure the alert threshold and alert notification settings. For more information, see the Configure monitoring and alerting when you create a DTS task section of the Configure monitoring and alerting topic.

7. Save the task settings and run a precheck.

   1. To view the parameters to be specified when you call the relevant API operation to configure the DTS task, move the pointer over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

   2. If you do not need to view or have viewed the parameters, click Next: Save Task Settings and Precheck in the lower part of the page.

   Note

   1. Before you can start the data migration task, DTS performs a precheck. You can start the data migration task only after the task passes the precheck.

   2. If the task fails to pass the precheck, click View Details next to each failed item. After you analyze the causes based on the check results, troubleshoot the issues. Then, run a precheck again.

   3. If an alert is triggered for an item during the precheck:

      1. If an alert item cannot be ignored, click View Details next to the failed item and troubleshoot the issues. Then, run a precheck again.

      2. If the alert item can be ignored, click Confirm Alert Details. In the View Details dialog box, click Ignore. In the message that appears, click OK. Then, click Precheck Again to run a precheck again. If you ignore the alert item, data inconsistency may occur, and your business may be exposed to potential risks.

8. Purchase an instance.

   1. Wait until the Success Rate becomes 100%. Then, click Next: Purchase Instance.

   2. On the Purchase Instance page, configure the Instance Class parameter for the data migration instance. The following table describes the parameters.

      Section	Parameter	Description

      Section	Parameter	Description
      New Instance Class	Resource Group	The resource group to which the data migration instance belongs. Default value: default resource group. For more information, see What is Resource Management?
      	Instance Class	DTS provides instance classes that vary in the migration speed. You can select an instance class based on your business scenario. For more information, see Instance classes of data migration instances.

   3. Read and agree to Data Transmission Service (Pay-as-you-go) Service Terms by selecting the check box.

   4. Click Buy and Start. In the message that appears, click OK.

      You can view the progress of the task on the Data Migration page.

Example of adding column mapping to HBase table

1. Add column mapping to the table created by HBase.

   ALTER TABLE test MAP DYNAMIC COLUMN f:_mongo_id_ HSTRING/HINT/..., person:name HSTRING, person:age HINT;

2. Add a secondary index to the table created by HBase.

   CREATE INDEX idx ON test(f:_mongo_id_);

HBase table migration example (ETL)

MongoDB Document

{
  "_id" : 0,
  "person" : {
    "name" : "cindy0",
    "age" : 0,
    "student" : true
  }
}

ETL Data Processing Statement

script:e_expand_bson_value("*", "_id")

Migration Result

Example of assignment configuration

Source MongoDB Data Structure

{
  "_id":"62cd344c85c1ea6a2a9f****",
  "person":{
    "name":"neo",
    "age":"26",
    "sex":"male"
  }
}

Target Lindorm Table Structure

New Column Configuration