Data Transmission Service:Configure two-way data synchronization between ApsaraDB for MongoDB replica set instances

How it works

A two-way synchronization instance consists of two tasks:

1. Forward task: Synchronizes data from Instance A to Instance B. Handles schema synchronization, full data synchronization, and incremental data synchronization.

2. Reverse task: Synchronizes data from Instance B back to Instance A. Handles incremental data synchronization only.

Configure forward task --> Precheck --> Purchase --> Forward task Running
                                                         |
                                       Configure reverse task --> Precheck --> Reverse task Running
                                                                                     |
                                                                   Both tasks Running = Two-way sync active

Prerequisites

Before you begin, make sure that:

• Source and destination ApsaraDB for MongoDB replica set instances are created. For more information, see Create a replica set instance. For supported database versions, see Overview of data synchronization scenarios.

• The destination instance has at least 10% more available storage than the total data size in the source instance.

• The replication.oplogGlobalIdEnabled parameter is set to true on both the source and destination instances. For more information, see Configure database parameters for an instance.

Step 1: Open the data synchronization page

Use one of the following methods to navigate to the Data Synchronization page:

DTS console

1. Log on to the DTS console.

2. In the left-side navigation pane, click Data Synchronization.

3. In the upper-left corner, select the region where the data synchronization task resides.

DMS console

1. Log on to the DMS console.

2. In the top navigation bar, move the pointer over Data + AI and choose DTS (DTS) > Data Synchronization.

3. From the drop-down list to the right of Data Synchronization Tasks, select the region where the data synchronization instance resides.

Step 2: Create and configure the forward synchronization task

1. Click Create Task.

2. (Optional) Click New Configuration Page in the upper-right corner.

   Note

   Skip this step if the Back to Previous Version button is displayed. The new version of the configuration page is recommended.

3. Configure the source and destination databases with the following parameters.

   Source database

   Parameter	Description
   Task Name	DTS automatically generates a name. Specify a descriptive name to identify the task easily. A unique name is not required.
   Select Existing Connection	If the database instance is registered with DTS, select it from the drop-down list. DTS populates the remaining parameters automatically. For more information, see Manage database connections. If the instance is not registered, configure the parameters below.
   Database Type	Select MongoDB.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	Select the region of the source ApsaraDB for MongoDB instance.
   Replicate Data Across Alibaba Cloud Accounts	Select No.
   Architecture	Select Replica Set.
   Migration Method	Select Oplog.
   Instance ID	Select the source ApsaraDB for MongoDB instance.
   Authentication Database	The database that stores account credentials. Default: admin.
   Database Account	The database account of the source instance. The account must have read permissions on the source database, the config database, the admin database, and the local database. For more information, see Use DMS to manage database accounts.
   Database Password	The password for the database account.
   Encryption	Select Non-encrypted, SSL-encrypted, or Mongo Atlas SSL based on your requirements. Available options depend on the Access Method and Architecture settings. If Architecture is Sharded Cluster and Migration Method is Oplog, SSL-encrypted is unavailable.

   Destination database

   Parameter	Description
   Select Existing Connection	If the database instance is registered with DTS, select it from the drop-down list. DTS populates the remaining parameters automatically. For more information, see Manage database connections. If the instance is not registered, configure the parameters below.
   Database Type	Select MongoDB.
   Access Method	Select Alibaba Cloud Instance.
   Instance Region	Select the region of the destination ApsaraDB for MongoDB instance.
   Replicate Data Across Alibaba Cloud Accounts	Select No.
   Architecture	Select Replica Set.
   Instance ID	Select the destination ApsaraDB for MongoDB instance.
   Authentication Database	The database that stores account credentials. Default: admin.
   Database Account	The database account of the destination instance. The account must have dbAdminAnyDatabase permission, read and write permissions on the destination database, and read permissions on the local database. For more information, see Use DMS to manage database accounts.
   Database Password	The password for the database account.
   Encryption	Select Non-encrypted, SSL-encrypted, or Mongo Atlas SSL based on your requirements. Available options depend on the Access Method and Architecture settings. If the destination is an ApsaraDB for MongoDB instance with Architecture set to Sharded Cluster, SSL-encrypted is unavailable.

4. Click Test Connectivity and Proceed.

   Note

   Make sure that the CIDR blocks of DTS servers are added to the security settings of both the source and destination databases. For more information, see Add the CIDR blocks of DTS servers.

Step 3: Configure synchronization objects

1. In the Configure Objects step, configure the following parameters.

   Parameter	Description
   Synchronization Types	Incremental Data Synchronization is selected by default. Also select Schema Synchronization and Full Data Synchronization.
   Processing Mode of Conflicting Tables	Precheck and Report Errors: Checks for collections with identical names in the source and destination. If identical names exist, an error is returned and the task cannot start. To resolve conflicts, use object name mapping to rename collections. Ignore Errors and Proceed: Skips the identical-name check. If the destination has a record with the same primary key or unique key value, DTS does not synchronize that record and retains the existing data. Data may fail to be initialized, only specific columns are synchronized, or the data synchronization instance fails.
   Synchronization Topology	Select Two-way Synchronization.
   Exclude DDL Operations	Yes: Excludes DDL operations. No: Synchronizes DDL operations. To maintain stability, DDL operations are synchronized only in the forward direction.
   Conflict Resolution Policy	Set to Ignore (fixed value). Conflict records in the destination database are retained. For more information, see Conflict resolution.
   Capitalization of Object Names in Destination Instance	Default: DTS default policy. Select other options to match the capitalization of object names in the source or destination database. For more information, see Specify the capitalization of object names in the destination instance.
   Source Objects	Select databases or collections from the Source Objects panel and click the right-arrow icon to add them to the Selected Objects panel.
   Selected Objects	Right-click an object to rename it in the destination database or map it to a specific destination object. For more information, see Map object names. To remove an object, select it and click the left-arrow icon. To synchronize incremental data by databases or collections, right-click Selected Objects and select the operations in the dialog box that appears. To filter data during full data synchronization, right-click a collection and configure filter conditions. For more information, see Specify filter conditions. If you use object name mapping to rename a database or collection, dependent objects may fail to synchronize.

2. Click Next: Advanced Settings and configure the following parameters.

   Parameter	Description
   Dedicated Cluster for Task Scheduling	By default, the task runs on a shared cluster. To improve stability, purchase a dedicated cluster. For more information, see What is a DTS dedicated cluster.
   Retry Time for Failed Connections	Time range for retrying failed connections. Valid values: 10 to 1440 minutes. Default: 720. Set this to a value greater than 30. If DTS reconnects within the specified time, the task resumes. Otherwise, the task fails. If multiple tasks share a source or destination database, the shortest retry time takes precedence. DTS charges for the instance during retry.
   Retry Time for Other Issues	Time range for retrying failed DDL or DML operations. Valid values: 1 to 1440 minutes. Default: 10. Set this to a value greater than 10. This value must be smaller than Retry Time for Failed Connections.
   Enable Throttling for Full Data Synchronization	Configure Queries per second (QPS) to the source database, RPS of Full Data Migration, and Data migration speed for full migration (MB/s) to reduce load on the destination server. Available only when Full Data Synchronization is selected.
   Only one data type for primary key _id in a table of the data to be synchronized	Whether the data type for primary key _id is unique in each collection. Yes: DTS does not scan the _id data type and synchronizes only one data type per collection. No: DTS scans the _id data type and synchronizes all matching data. Set based on your data. Displayed only when Full Data Synchronization is selected.
   Enable Throttling for Incremental Data Synchronization	Configure RPS of Incremental Data Synchronization and Data synchronization speed for incremental synchronization (MB/s) to reduce load on the destination server.
   Environment Tag	An environment tag to identify the DTS instance. Optional.
   Configure ETL	Yes: Configure extract, transform, and load (ETL). For more information, see What is ETL? and Configure ETL in a data synchronization task. No: Do not configure ETL.
   Monitoring and Alerting	Yes: Configure alert thresholds and notification settings for task failures or synchronization latency. For more information, see Configure monitoring and alerting. No: Do not enable alerting.

3. Click Next Step: Data Verification and configure data verification. For more information, see Configure a data verification task.

Step 4: Run the precheck and purchase the instance

1. Save the task settings and run a precheck.

   • To view API parameters for the task, hover over Next: Save Task Settings and Precheck and click Preview OpenAPI parameters.

   • Click Next: Save Task Settings and Precheck.

   Note

   • DTS performs a precheck before starting the task. The task can start only after it passes the precheck.

   • If the precheck fails, click View Details next to the failed item, troubleshoot the issue, and rerun the precheck.

   • If an alert is triggered: for non-ignorable items, troubleshoot and rerun the precheck. For ignorable items, click Confirm Alert Details, then Ignore, then OK, and rerun the precheck. Ignoring alerts may cause data inconsistency.

2. Wait until the Success Rate reaches 100%, then click Next: Purchase Instance.

3. On the purchase page, configure the instance.

   Parameter	Description
   Billing Method	Subscription: Pay upfront for long-term use. Pay-as-you-go: Billed hourly. Suitable for short-term use. Release the instance when no longer needed.
   Resource Group Settings	The resource group for the instance. Default: default resource group. For more information, see What is Resource Management?
   Instance Class	Select an instance class based on synchronization speed requirements. For more information, see Instance classes of data synchronization instances.
   Subscription Duration	Available for Subscription only. Valid values: 1 to 9 months, or 1, 2, 3, or 5 years.

4. Read and select Data Transmission Service (Pay-as-you-go) Service Terms.

5. Click Buy and Start. In the dialog box, click OK. The forward synchronization task appears in the task list.

Step 5: Configure the reverse synchronization task

1. Wait until the initial synchronization completes and the forward synchronization task status changes to Running.

2. In the task list, find the reverse synchronization task and click Configure Task.

3. Configure the reverse synchronization task by repeating the configuration sub-steps in Step 2 (from sub-step 3 onward), Step 3, and the precheck in Step 4, with the following adjustments:

   Note

   The reverse task is automatically created when you purchase the forward task. You do not need to click Create Task (Step 2, sub-step 1) or purchase a new instance (Step 4, sub-step 3 onward).

   Important

   - The source instance in the reverse direction is the destination instance in the forward direction, and vice versa. Make sure that database names, accounts, and passwords are configured accordingly. - Do not use object name mapping in the reverse direction. Otherwise, data inconsistency may occur. - Instance Region cannot be modified in the reverse direction. Configure only the parameters displayed in the console. - For Processing Mode of Conflicting Tables, make sure that collections already synchronized to the destination by the forward task are ignored. - You cannot select the same objects from the Selected Objects list used by the forward task. - The reverse task ignores DDL operations.

4. Wait until the Success Rate reaches 100%, then click Back.

Step 6: Verify the synchronization status

After both the forward and reverse synchronization tasks enter the Running state, two-way data synchronization is active. Monitor the task status in the task list.

Billing

Limitations