This topic provides answers to frequently asked questions (FAQ) about Data Transmission Service (DTS). If DTS returns error messages, you can refer to the Common Errors and Troubleshooting topic to troubleshoot the errors. If you do not receive error messages, you can identify the causes of the errors and fix the errors based on the FAQ in this topic.
FAQ types
FAQ are classified into the following types:
How am I charged for DTS?
You are charged for DTS based on the subscription and pay-as-you-go billing methods. For more information about billing methods, see Billing overview.
How do I view the bills of DTS?
For more information about how to view the bills of DTS, see View bills.
Am I still charged during the period in which a DTS instance is paused?
You are not charged during the period in which a data migration instance is paused. You are still charged during the period in which a data synchronization instance is paused if Incremental Data Synchronization is selected when you configure the data synchronization task. This is because DTS only stops writing data to the destination database and continues to collect data on incremental changes in the source database.
Why is the price of data synchronization higher than that of data migration?
Data synchronization comes with more advanced features. For example, you can reselect the objects to be synchronized, and configure two-way synchronization between MySQL databases. In addition, data synchronization ensures low network latency by transmitting data over an internal network.
How do I release a subscription instance before it expires?
You can release your subscription instance after you change the billing method of the instance from subscription to pay-as-you-go. For more information about how to change the billing method, see Change the billing method.
Am I able to change the billing method of a task from subscription to pay-as-you-go?
Yes, you can change the billing method of a task from subscription to pay-as-you-go. For more information about how to change the billing method, see Change the billing method.
Am I able to change the billing method of a task from pay-as-you-go to subscription?
Yes, you can change the billing method of a data synchronization task or a change tracking task from pay-as-you-go to subscription. For more information about how to change the billing method, see Change the billing method.
Note
The billing method of data migration tasks can only be pay-as-you-go.
Why am I charged for a DTS instance that was previously free?
This is because the free trial period of your DTS instance elapses. DTS provides a free trial period for a task whose destination database is an ApsaraDB database. You are charged for the task after the free trial period elapses.
Why am I still charged for a DTS instance that has been released?
DTS generates bills on a daily basis for pay-as-you-go DTS instances. You are charged for your DTS instance on the day when the instance is released because DTS is used on that day.
How am I charged for a pay-as-you-go DTS instance?
You are charged for a pay-as-you-go DTS instance only when an incremental task is in progress, including the period in which incremental data synchronization is paused, but excluding the period in which incremental data migration is paused. For more information, see Billing methods.
FAQ about performance and instance types
Am I able to upgrade a DTS instance?
Yes, you can upgrade a DTS instance. To upgrade a DTS instance, perform the following operations: Log on to the DTS console. Find the instance that you want to upgrade, click the 
icon in the Actions column, and then select Upgrade.
Important
After you upgrade a DTS instance, you cannot downgrade the DTS instance.
Am I able to downgrade a DTS instance?
No, DTS does not allow you to downgrade a DTS instance.
How long does it take to synchronize or migrate data?
The amount of time required for DTS tasks cannot be estimated. This is because the transmission performance of DTS is affected by various factors such as the internal payload of DTS, the payload of source and destination databases, the volume of data to be transmitted, whether incremental data migration or synchronization is involved in DTS tasks, and network conditions. If you require high performance, we recommend that you select instance types that provide better performance. For more information about instance types, see Instance classes of data migration instances and Instance classes of data synchronization instances.
Why am I unable to find a specific DTS instance in the DTS console?
If the DTS instance is a subscription instance, the instance may have been released due to expiration.
The resource group is invalid. We recommend that you select All Resources.
The region is invalid. Check whether the region that you select is the region in which the DTS instance resides.
The task type is invalid. Check whether you are accessing the instance list page that displays instances of the required task type. For example, a synchronization instance is displayed only on the Data Synchronization Tasks page.
The instance is released due to expiration or overdue payments. If a subscription instance expires or an overdue payment occurs, the instance stops. If you do not settle the overdue payment within seven days after the service stops, the system releases and deletes the instance. For more information, see Expiration and overdue payments.
Why is an alert generated for the check item that is related to the data eviction policy of Redis?
If the data eviction policy (maxmemory-policy) of the destination database is not set to noeviction, data inconsistency may occur between the source and destination databases. For more information about data eviction policies, see What is the default eviction policy of Tair?
What do I do if the items that are related to binary logs fail to pass the precheck for an incremental data migration task?
Check whether the binary logs of the source database are normal. For more information, see Precheck on the binary logging of the source database.
FAQ about database connections
What do I do if the connection to the source database fails?
Check whether the information and settings of the source database are normal. For more information, see Source database connectivity.
What do I do if the connection to the destination database fails?
Check whether the information and settings of the destination database are normal. For more information, see Destination database connectivity.
How do I migrate or synchronize data from or to database instances that reside in regions not supported by DTS?
For a data migration task, you can apply for a public endpoint for a database instance, such as an ApsaraDB RDS for MySQL instance. This way, the database instance can be connected by using a public IP address. Then, add the CIDR blocks of DTS servers that reside in a region supported by DTS to the whitelist of the database instance. For more information about the CIDR blocks of DTS servers that need to be added to the whitelist of the database instance, see Add the CIDR blocks of DTS servers.
For a data synchronization task, you cannot synchronize data from or to database instances in regions that are not supported by DTS because database instances cannot be connected by using a public IP address during data synchronization.
FAQ about data synchronization
What types of databases does DTS support for data synchronization?
DTS supports data synchronization between various data sources, such as relational database management system (RDBMS) databases, NoSQL databases, and online analytical processing (OLAP) databases. For more information about the database instances that DTS supports for data synchronization, see Overview of data synchronization scenarios.
What are the differences between data migration and data synchronization?
The following table describes the differences between data migration and data synchronization.
Item | Data migration | Data synchronization |
Item | Data migration | Data synchronization |
Use scenarios | The data migration feature is used to migrate data from on-premises databases, self-managed databases hosted on Elastic Compute Service (ECS) instances, and databases on third-party clouds to Alibaba Cloud. | The data synchronization feature is used to synchronize data between two data sources in real time. This feature is suitable for scenarios such as active geo-redundancy, disaster recovery, cross-border data synchronization, query load balancing, cloud business intelligence (BI) systems, and real-time data warehousing. |
Supported database services | For more information, see Overview of data migration scenarios. | For more information, see Overview of data synchronization scenarios. |
Supported deployment environments of databases | Alibaba Cloud database instance Self-managed database with a public IP address Self-managed database that is connected over Database Gateway (DG) Self-managed database that is connected over Cloud Enterprise Network (CEN) Self-managed database that is hosted on an ECS instance Self-managed database that is connected over Express Connect, VPN Gateway, or Smart Access Gateway
| Alibaba Cloud database instance Self-managed database that is connected over DG Self-managed database that is connected over CEN Self-managed database that is hosted on an ECS instance Self-managed database that is connected over Express Connect, VPN Gateway, or Smart Access Gateway
Note The data synchronization feature ensures low network latency by transmitting data over an internal network. |
Features | You can enable the object name mapping feature for columns, tables, and databases. You can filter the data to be migrated. You can select the types of SQL operations to be migrated. For example, you can migrate only INSERT operations. You can read and migrate data across self-managed databases that are deployed in virtual private clouds (VPCs) owned by different Alibaba Cloud accounts.
| You can enable the object name mapping feature for columns, tables, and databases. You can filter the data to be synchronized. You can modify the objects to be synchronized. You can configure two-way synchronization between databases such as MySQL databases. You can select the types of SQL operations to be synchronized. For example, you can synchronize only INSERT operations.
|
Billing methods | Only the pay-as-you-go billing method is supported. | The pay-as-you-go and subscription billing methods are supported. |
Billing rules | You are charged only when incremental data migration is in progress, excluding the period in which incremental data migration is paused. You are not charged for schema migration or full data migration. | If you use the pay-as-you-go billing method, you are charged only when incremental data synchronization is in progress, including the period in which incremental data synchronization is paused. You are not charged for schema synchronization or full data synchronization. If you use the subscription billing method, the fee is deducted at the time of purchase based on the quantity and configurations that you specify.
|
Note
For databases that are not supported by the data synchronization feature, such as databases in ApsaraDB for MongoDB instances, you can perform incremental data migration to synchronize data from or to the databases.
How does DTS work in data synchronization mode?
For more information about how DTS works in data synchronization mode, see Architecture.
How do I calculate a synchronization latency?
A synchronization latency is the difference between the timestamp of the latest data record synchronized to the destination database and the current timestamp of the source database. Unit: millisecond.
Note
In most cases, the latency is less than 1,000 milliseconds.
Am I able to add tables to a data synchronization task?
Yes, you can add tables to a data synchronization task. For more information about how to add tables to a data synchronization task, see Add an object to a data synchronization task.
How do I modify the objects such as tables and fields in a data synchronization task in the Running state?
When the full data synchronization phase of a data synchronization task ends, you can modify the objects to be synchronized in the incremental data synchronization phase. For more information about how to modify the objects to be synchronized in a data synchronization task in the Running state, see Add an object to a data synchronization task and Remove an object from a data synchronization task.
Does data inconsistency occur if a data synchronization task is resumed after it is paused for a period of time?
If the source database changes during the suspension of a data synchronization task, data in the source and destination databases may become inconsistent. After you resume the data synchronization task and synchronize incremental data to the destination database, data in the source and destination databases is consistent.
If data in the source database of an incremental data synchronization task is deleted, is the synchronized data in the destination database deleted?
If you do not select delete for the DML operations to be synchronized in the incremental data synchronization task, the data in the destination database is not deleted. Otherwise, the synchronized data in the destination database is deleted.
Is the data of the destination Redis instance overwritten during the synchronization between Redis instances?
Data with the same key is overwritten. The destination database is checked in the precheck phase. If the destination database is not empty, an error message is returned.
Am I able to filter fields or data in a data synchronization task?
Yes, you can filter fields or data in a data synchronization task. You can specify an SQL WHERE condition to filter the data to be synchronized. For more information about how to filter fields or data, see Set filter conditions.
Am I able to convert a data synchronization task to a data migration task?
No, different types of tasks cannot be converted to each other.
Am I able to synchronize only data without synchronizing schemas?
Yes, you can synchronize only data without synchronizing schemas. To do so, do not select Schema Synchronization when you configure a data synchronization task.
What are the possible causes for data inconsistency between the source and destination databases in a data synchronization task?
Data inconsistency may occur due to the following causes:
You do not clear the data in the destination database when you configure the task, and the destination database stores historical data.
You select only the incremental data synchronization module when you configure the task, and you do not select the full data synchronization module.
You select only the full data synchronization module when you configure the task, and you do not select the incremental data synchronization module. The data in the source database changes after the task is complete.
You use tools other than DTS to write data to the destination database.
Incremental data has not been written to the destination database due to the incremental write latency.
Am I able to modify the names of the synchronized objects in the destination database in a data synchronization task?
Yes, you can modify the names of the synchronized objects in the destination database when you configure a data synchronization task. For more information about how to modify the names of the synchronized objects in the destination database, see Rename an object to be synchronized.
Am I able to synchronize DML or DDL operations?
Yes, you can synchronize DML and DDL operations between relational databases. The supported DML operations are INSERT, UPDATE, and DELETE. The supported DDL operations are CREATE, DROP, ALTER, RENAME, and TRUNCATE.
Can a read-only database instance be used as the source instance for a data synchronization task?
If incremental data does not need to be synchronized in a data synchronization task, a read-only database instance can be used as the source instance. Otherwise, take note of the following items:
If the read-only database instance records transaction logs, such as an instance of ApsaraDB RDS for MySQL 5.7 or 8.0, it can be used as the source instance.
If the read-only database instance does not record transaction logs, such as an instance of ApsaraDB RDS for MySQL 5.6, it cannot be used as the source instance.
Am I able to synchronize database shards or table partitions?
Yes, you can synchronize database shards and table partitions. For example, you can synchronize database shards and table partitions from a MySQL database or a PolarDB for MySQL cluster to an AnalyticDB for MySQL cluster. This allows you to merge multiple tables.
Why is the size of data in the destination database smaller than that in the source database after data synchronization is complete?
If the data to be synchronized is filtered or the source database has many table partitions, the size of data in the destination database may be smaller than that in the source database after data synchronization is complete.
Is two-way synchronization supported by a cross-account data synchronization task?
You can configure a two-way synchronization task across Alibaba Cloud accounts only between ApsaraDB RDS for MySQL instances, between PolarDB for MySQL clusters, or between Tair instances.
Am I able to configure a cross-border two-way synchronization task?
No, you cannot configure a cross-border two-way synchronization task.
Why does the progress of an incremental data synchronization task never reach 100%?
The progress of an incremental data synchronization task cannot reach 100%. If DTS performs incremental data synchronization, DTS continuously synchronizes changes from the source database to the destination database in real time and can only be manually stopped. If real-time incremental synchronization is no longer needed, you can stop the synchronization task in the DTS console.
Why does some data fail to be synchronized in an incremental synchronization task?
If you configure only an incremental synchronization task for a DTS instance, DTS synchronizes only the incremental data after the task is started. The data that is stored in the source database before the task is started is not synchronized to the destination database. We recommend that you select Incremental Data Synchronization, Schema Synchronization, and Full Data Synchronization when you configure a data synchronization task. This ensures data consistency.
Is the performance of the source ApsaraDB RDS database affected when I synchronize full data from the source database?
Yes, the query performance of the source ApsaraDB RDS database is affected when you synchronize full data from the source database. To reduce the impact on the performance of the source database, use one of the following methods:
Upgrade the specifications of your source instance.
Suspend the DTS task first and resume the task after the load on the source database decreases.
Reduce the synchronization or migration rate of the DTS task. For more information about how to adjust the migration rate, see Enable throttling for full data migration.
Does DTS support data synchronization from a MySQL database to an ApsaraDB for ClickHouse cluster?
DTS has discontinued the feature of synchronizing data from a MySQL database to an ApsaraDB for ClickHouse cluster. If you have already purchased a DTS instance to synchronize data from a MySQL database to an ApsaraDB for ClickHouse cluster, the DTS instance is not affected. For more information about this feature, see Synchronize data from an ApsaraDB RDS for MySQL instance to an ApsaraDB for ClickHouse cluster in the documentation of DTS.
Why is the latency not displayed for a data synchronization instance whose source database is a PolarDB-X 1.0 database?
A DTS task whose source database is a PolarDB-X 1.0 database is a distributed task. The metrics that are monitored by DTS exist only in subtasks. Therefore, no latency is displayed for the DTS instance whose source database is a PolarDB-X 1.0 database. You can click the instance ID to go to the instance details page and view the latency of the instance on the Subtask Details tab of the Task Management page.
Why is the error DTS-071001 reported in a multi-table merging task?
The error occurs because you perform online DDL operations on the source database when the multi-table merging task is in progress. You modify objects such as table schemas in the source database, but do not modify the corresponding objects in the destination database.
What do I do if I am unable to configure a whitelist when I configure a task in the DTS console of the previous version?
You can configure tasks in the DTS console of the new version.
What do I do if a data synchronization task fails because a DDL operation is performed on the source database?
You can perform the same DDL operation on the destination database and restart the task. We recommend that you do not use tools such as pt-online-schema-change to perform DDL operations on objects during data synchronization. Otherwise, data synchronization may fail. If you use only DTS to write data to the destination database, you can use Data Management (DMS) to perform online DDL operations on source tables during data synchronization. Alternatively, you can remove the tables that are affected by DDL operations. For more information about how to remove the objects to be synchronized, see Remove an object from a data synchronization task.
What do I do if a data synchronization task fails because a DDL operation is performed on the destination database?
If a destination database or a table of the destination database is deleted during incremental data synchronization, the data synchronization task fails. In this case, you can use one of the following methods to resume the task:
Method 1: Reconfigure the task. Do not select the database or table that causes the task failure as the object to be synchronized.
Method 2: Modify the objects to be synchronized. Remove the database or table that causes the task failure from the data synchronization task. For more information, see Remove an object from a data synchronization task.
Am I able to resume a data synchronization task after it is released? Can data consistency be ensured if I reconfigure the task?
A data synchronization task cannot be resumed after it is released. When you reconfigure the data synchronization task, if you do not select Full Data Synchronization, the incremental data generated after the task is released and before the new task is started cannot be synchronized to the destination database. In this case, data consistency cannot be ensured. If your business requires high accuracy, you can delete the synchronized data from the destination database and reconfigure the data synchronization task. When you reconfigure the task, select Schema Synchronization and Full Data Synchronization for the Task Stages parameter. By default, Incremental Data Synchronization is selected.
What do I do if the progress of a full data synchronization task remains unchanged for an extended period of time?
If the tables to be synchronized do not contain a primary key, full data synchronization is very slow. We recommend that you define a primary key for each table to be synchronized from the source database before you start the data synchronization task.
Am I able to synchronize only the data that does not exist in the destination table from the source table of the same name?
Yes, you can synchronize only the data that does not exist in the destination table from the source table of the same name. When you configure a DTS task, you can set the Processing Mode of Conflicting Tables parameter to Ignore Errors and Proceed. During full data synchronization, if the source and destination databases have the same schemas, and a data record has the same primary key value as an existing data record in the destination database, this record is not synchronized to the destination database.
How do I configure a data synchronization task across Alibaba Cloud accounts?
You must use the Alibaba Cloud account to which the source instance belongs to configure Resource Access Management (RAM) authorization, and then use the Alibaba Cloud account to which the destination instance belongs to configure a DTS task. For more information, see Configure a DTS task across Alibaba Cloud accounts.
What do I do if I am unable to select a DMS logical database instance?
Make sure that you select the region in which the logical database instance resides. If the issue persists, maybe only one instance exists and the instance is selected by default. In this case, continue to configure other parameters.
Am I able to synchronize functions in a data synchronization task if the source database is an SQL Server database?
No, you cannot synchronize functions in a data synchronization task if the source database is an SQL Server database. If you select tables as the objects to be synchronized, DTS also does not synchronize other objects such as views, triggers, or stored procedures to the destination database.
What do I do if an error is reported in a data synchronization task?
You can refer to Common errors and troubleshooting to view the detailed solution based on the error message.
What do I do if I want to enable the hotspot table merging feature for a data synchronization task?
You can set the trans.hot.merge.enable parameter to true. For more information, see Modify the parameters of a DTS instance.
Am I able to synchronize a system database such as the sys database?
No, you cannot synchronize a system database such as the sys database.
Am I able to synchronize data from or to the admin or local database in a MongoDB instance?
No, you cannot synchronize data from or to the admin or local database in a MongoDB instance.
When am I able to configure a two-way synchronization task in the reverse direction?
You can configure a two-way synchronization task in the reverse direction only if the incremental synchronization task in the forward direction is not delayed.
Am I able to scale out or scale in the nodes of the source PolarDB-X 1.0 instance in a data synchronization task?
No, you cannot scale out or scale in the nodes of the source PolarDB-X 1.0 instance in a DTS task. If the nodes of the source PolarDB-X 1.0 instance are scaled out or scaled in, you must reconfigure the task.
Is the data synchronized from DTS to Kafka unique?
No, DTS cannot ensure that the data synchronized or migrated to Kafka is unique. This is because the data written to Kafka is appended, and duplicate data may exist when a DTS task is restarted or logs in the source database are repeatedly pulled. DTS ensures the idempotence of data. Data is sorted in order based on the time when data is written, and the latest values of duplicate data are appended at the back.
Why is Full Data Synchronization not displayed when I synchronize data between Redis instances?
Incremental Data Synchronization is displayed. If you select this option for data synchronization between Redis instances, both full data synchronization and incremental data synchronization are performed.
Can I skip full data synchronization?
Yes, you can skip full data synchronization. After you skip full data synchronization, incremental data synchronization continues. However, an error may occur. We recommend that you do not skip full data synchronization.
Am I able to configure a data synchronization task that automatically starts at the specified point in time?
No, you cannot configure a data synchronization task that automatically starts at the specified point in time.
Are the fragmented spaces in a table synchronized during synchronization?
No, the fragmented spaces in a table are not synchronized.
What items do I need to take note of when I synchronize data from a MySQL 8.0 instance to a MySQL 5.6 instance?
You must create a database in the MySQL 5.6 instance before you start the data synchronization task. To ensure compatibility, the version of the destination instance must be the same as or later than that of the source database. If the version of the destination instance is earlier than that of the source database, database compatibility issues may occur.
Am I able to synchronize accounts from the source database to the destination database?
You can synchronize accounts only between ApsaraDB RDS for MySQL instances.
Am I able to configure a two-way synchronization task across Alibaba Cloud accounts?
You can configure a two-way synchronization task across Alibaba Cloud accounts only between ApsaraDB RDS for MySQL instances, between PolarDB for MySQL clusters, or between Tair instances.
Note
For tasks that do not have the Replicate Data Across Alibaba Cloud Accounts configuration item, you can use CEN to implement two-way synchronization. For more information, see the "Connect databases to DTS across Alibaba Cloud accounts or regions" section of the Connect an on-premises database to DTS by using CEN topic.
Does the migrated data exist in the source database after a data migration task is complete?
Yes, the migrated data exists in the source database after a data migration task is complete. DTS copies data from the source database to the destination database during data migration and synchronization. The data in the source database is not affected.
What types of databases does DTS support for data migration?
DTS supports data migration between various data sources, such as RDBMS databases, NoSQL databases, and OLAP databases. For more information, see Overview of data migration scenarios.
How does DTS work in data migration mode?
For more information about how DTS works in data migration mode, see Architecture.
Am I able to modify the objects to be migrated in a data migration task?
No, you cannot modify the objects to be migrated in a data migration task.
Am I able to add tables to a data migration task?
No, you cannot add tables to a data migration task.
How do I modify the objects such as tables and fields in a data migration task in the Running state?
DTS does not allow you to modify the objects to be migrated in a data migration task.
Does data inconsistency occur if a data migration task is restarted after it is paused for a period of time?
If the source database changes during the suspension of a data migration task, data in the source and destination databases may become inconsistent. After you restart the data migration task and migrate incremental data to the destination database, data in the source and destination databases is consistent.
Am I able to convert a data migration task to a data synchronization task?
No, different types of tasks cannot be converted to each other.
Am I able to migrate only data without migrating schemas?
Yes, you can migrate only data without migrating schemas. To do so, do not select Schema Migration when you configure a data migration task.
What are the possible causes for data inconsistency between the source and destination databases in a data migration task?
Data inconsistency may occur due to the following causes:
You do not clear the data in the destination database when you configure the task, and the destination database stores historical data.
You select only the incremental data migration module when you configure the task, and you do not select the full data migration module.
You select only the full data migration module when you configure the task, and you do not select the incremental data migration module. The data in the source database changes after the task is complete.
You use tools other than DTS to write data to the destination database.
Incremental data has not been written to the destination database due to the incremental write latency.
Am I able to modify the names of the migrated objects in the destination database in a data migration task?
Yes, you can modify the names of the migrated objects in the destination database when you configure a data migration task. For more information about how to modify the names of the migrated objects in the destination database, see Object name mapping.
Am I able to migrate DML or DDL operations?
Yes, you can migrate DML and DDL operations between relational databases. The supported DML operations are INSERT, UPDATE, and DELETE. The supported DDL operations are CREATE, DROP, ALTER, RENAME, and TRUNCATE.
Can a read-only database instance be used as the source instance for a data migration task?
If incremental data does not need to be migrated in a data migration task, a read-only database instance can be used as the source instance. Otherwise, take note of the following items:
If the read-only database instance records transaction logs, such as an instance of ApsaraDB RDS for MySQL 5.7 or 8.0, it can be used as the source instance.
If the read-only database instance does not record transaction logs, such as an instance of ApsaraDB RDS for MySQL 5.6, it cannot be used as the source instance.
Am I able to migrate database shards or table partitions?
Yes, you can migrate database shards and table partitions. For example, you can migrate database shards and table partitions from a MySQL database or a PolarDB for MySQL cluster to an AnalyticDB for MySQL cluster. This allows you to merge multiple tables.
Am I able to filter fields or data in a data migration task?
Yes, you can filter fields or data in a data migration task. You can specify an SQL WHERE condition to filter the data to be migrated. For more information about how to filter fields or data, see Filter the data to be migrated.
Why is the size of data in the destination database smaller than that in the source database after data migration is complete?
If the data to be migrated is filtered or the source database has many table partitions, the size of data in the destination database may be smaller than that in the source database after data migration is complete.
Why does the number of migrated objects exceed the total number of objects in a data migration task?
The total number of objects displayed is an estimated value. After the data migration task is complete, the total number is adjusted to an exact value.
Why is the increment_trx table created in the destination database during data migration?
The increment_trx table is an offset table created by DTS in the destination database. This table records the offsets of incremental data migration and can be used to resolve a resumable upload issue if the task is abnormally restarted. Do not delete the increment_trx table during data migration. Otherwise, data migration may fail.
Is resumable upload supported by a data migration task in the full data migration phase?
Yes, you can resume a data migration task from the point at which it is interrupted in the full data migration phase. If a data migration task is restarted after it is paused for a period in the full data migration phase, the task continues to migrate data from the offset where the migration is complete. You do not need to start data migration from the beginning.
Am I able to pause a data migration task in the full data migration phase before the task is complete?
Yes, you can pause a data migration task in the full data migration phase before the task is complete.
How do I migrate partial data from an ApsaraDB RDS for MySQL instance to a self-managed MySQL database?
When you configure a data migration task, you can select the objects to be migrated in the Source Objects section or filter objects in the Selected Objects section based on your business requirements. The operations to migrate data between MySQL databases are similar. For more information, see Migrate data from a self-managed MySQL database to an ApsaraDB RDS for MySQL instance.
How do I migrate data between ApsaraDB RDS instances within the same Alibaba Cloud account?
DTS supports data migration and synchronization between ApsaraDB RDS instances. For more information, see Overview of data migration scenarios.
After a data migration task is started, an IOPS alert is reported for the source database. How do I ensure the stability of the source database in this case?
When a DTS task is in progress, you can use one of the following methods to reduce the impact of the DTS task on the source database if the load of the source database is high:
Upgrade the specifications of your source instance.
Suspend the DTS task first and resume the task after the load on the source database decreases.
Reduce the synchronization or migration rate of the DTS task. For more information about how to adjust the migration rate, see Enable throttling for full data migration.
Why am I unable to select a database named test for a data migration task?
A system database cannot be migrated. You can select a self-managed database for migration.
Why is the latency not displayed for a data migration instance whose source database is a PolarDB-X 1.0 database?
A DTS task whose source database is a PolarDB-X 1.0 database is a distributed task. The metrics that are monitored by DTS exist only in subtasks. Therefore, no latency is displayed for the DTS instance whose source database is a PolarDB-X 1.0 database. You can click the instance ID to go to the instance details page and view the latency of the instance on the Subtask Details tab of the Task Management page.
Why am I unable to migrate data from a MongoDB instance?
This is because the database to be migrated is the local or admin database. You cannot migrate data from or to the local or admin database in a MongoDB instance.
Why is the error DTS-071001 reported in a multi-table merging task?
The error occurs because you perform online DDL operations on the source database when the multi-table merging task is in progress. You modify objects such as table schemas in the source database, but do not modify the corresponding objects in the destination database.
What do I do if I am unable to configure a whitelist when I configure a task in the DTS console of the previous version?
You can configure tasks in the DTS console of the new version.
What do I do if a data migration task fails because a DDL operation is performed on the source database?
You can perform the same DDL operation on the destination database and restart the task. We recommend that you do not use tools such as pt-online-schema-change to perform online DDL operations on objects during data migration. Otherwise, data migration may fail. If you use only DTS to write data to the destination database, you can use DMS to perform online DDL operations during data migration.
What do I do if a data migration task fails because a DDL operation is performed on the destination database?
If a destination database or a table of the destination database is deleted during incremental data migration, the data migration task fails. You can reconfigure the task and do not select the database or table that causes the task failure as the object to be migrated.
Am I able to resume a data migration task after it is released? Can data consistency be ensured if I reconfigure the task?
A data migration task cannot be resumed after it is released. When you reconfigure the data migration task, if you do not select Full Data Migration, the incremental data generated after the task is released and before the new task is started cannot be migrated to the destination database. In this case, data consistency cannot be ensured. If your business requires high accuracy, you can delete the migrated data from the destination database and reconfigure the data migration task. When you reconfigure the task, select Schema Migration, Incremental Data Migration, and Full Data Migration for the Task Stages parameter.
What do I do if the progress of a full data migration task remains unchanged for an extended period of time?
If the tables to be migrated do not contain a primary key, full data migration is very slow. We recommend that you define a primary key for each table to be migrated from the source database before you start the data migration task.
Am I able to migrate only the data that does not exist in the destination table from the source table of the same name?
Yes, you can migrate only the data that does not exist in the destination table from the source table of the same name. When you configure a DTS task, you can set the Processing Mode of Conflicting Tables parameter to Ignore Errors and Proceed. During full data migration, if the source and destination databases have the same schemas, and a data record has the same primary key value as an existing data record in the destination database, this record is not migrated to the destination database.
How do I configure a data migration task across Alibaba Cloud accounts?
You must use the Alibaba Cloud account to which the source instance belongs to configure Resource Access Management (RAM) authorization, and then use the Alibaba Cloud account to which the destination instance belongs to configure a DTS task. For more information, see Configure a DTS task across Alibaba Cloud accounts.
What do I do if a data migration task fails and the error DTS-31008 is reported?
You can find the failed task in the DTS console and click View Cause to view details. For more information about the detailed solutions, see Common errors and troubleshooting.
What do I do if the connection to a self-managed database over Express Connect fails?
Check whether the CIDR blocks of DTS servers for connections over Express Connect in the required region are added to the IP address whitelist of the self-managed database. For more information about the CIDR blocks of DTS servers that need to be added to the whitelist of the self-managed database, see Whitelist DTS IP ranges for your user-created database.
Am I able to migrate functions in a data migration task if the source database is an SQL Server database?
No, you cannot migrate functions in a data migration task if the source database is an SQL Server database. If you select tables as the objects to be migrated, DTS also does not migrate other objects such as views, triggers, or stored procedures to the destination database.
What do I do if a full data migration task is run at a low speed?
This is because a large amount of data is to be migrated. Wait until the task is complete. You can log on to the DTS console and click the instance ID to go to the instance details page. On the Task Management page, click the Full Data Migration tab to view the progress of the full data migration task.
What do I do if an error is reported for schema migration?
You can log on to the DTS console and click the instance ID to go to the instance details page. On the Task Management page, click the Schema Migration tab to view the detailed error message and troubleshoot the error. For more information about how to troubleshoot common errors, see Common errors and troubleshooting.
Am I charged for schema migration or full data migration?
No, you are not charged for schema migration or full data migration. For more information, see Billable items.
Is the data of the ZSET type overwritten in the destination instance if I migrate data between Redis instances?
Yes, the data of the ZSET type is overwritten in the destination database if you migrate data between Redis instances. If the source and destination databases have the same key of the ZSET type, DTS deletes the data of the ZSET key from the destination database, and migrates all the members of the ZSET key from the source database to the destination database by running the ZADD command.
What is the impact of full data migration on the source database?
During full data migration, DTS divides data into slices and reads and writes data within the slicing range. The IOPS of the source database increases during data slicing. When DTS reads data within the slicing range, the IOPS, CachePool, and outbound bandwidth of the source database are affected. Based on the practical experience of DTS, these impacts can be ignored.
Am I able to scale out or scale in the nodes of the source PolarDB-X 1.0 instance in a data migration task?
No, you cannot scale out or scale in the nodes of the source PolarDB-X 1.0 instance in a DTS task. If the nodes of the source PolarDB-X 1.0 instance are scaled out or scaled in, you must reconfigure the task.
Is the data migrated from DTS to Kafka unique?
No, DTS cannot ensure that the data synchronized or migrated to Kafka is unique. This is because the data written to Kafka is appended, and duplicate data may exist when a DTS task is restarted or logs in the source database are repeatedly pulled. DTS ensures the idempotence of data. Data is sorted in order based on the time when data is written, and the latest values of duplicate data are appended at the back.
Does data inconsistency occur if I configure a full data migration task and then configure an incremental data migration task?
Yes, data inconsistency may occur. If you configure an incremental data migration task, data is migrated only after the incremental data migration task is started. Before the task is started, the incremental data that is generated in the source instance is not synchronized to the destination instance. If you need to migrate data without service interruption, select Schema Migration, Full Data Migration, and Incremental Data Migration when you configure a migration task.
Do I need to select Schema Migration when I configure an incremental data migration task?
If you select Schema Migration, DTS migrates the definitions of the objects to be migrated, such as the definition of Table A, to the destination instance before data is migrated. To perform incremental data migration, we recommend that you select Schema Migration, Full Data Migration, and Incremental Data Migration. This ensures data consistency during data migration.
Why is the storage space of the destination RDS instance larger than that of the source database when I migrate data from a self-managed database to an RDS instance?
DTS uses logical migration. DTS encapsulates the data to be migrated into SQL statements and then migrates the SQL statements to the destination RDS instance. In this case, binary logs are generated in the destination RDS instance. Therefore, the storage space of the destination RDS instance may be larger than that of the source database during the migration.
Am I able to select an ApsaraDB for MongoDB instance in a VPC as the source database to migrate data?
Yes, DTS allows you to select an ApsaraDB for MongoDB instance in a VPC as the source database to migrate data.
What is the impact of modifying the data of the source database when a data migration task is in progress?
If Schema Migration, Full Data Migration, and Incremental Data Migration are selected for the data migration task, DTS migrates all data changes from the source database to the destination database.
Is the source database affected if I release a data migration task that is complete?
No, the source database is not affected. After the state of the data migration task changes to Completed, you can release the data migration task.
Does DTS support incremental data migration for an ApsaraDB for MongoDB instance?
Yes, DTS supports incremental data migration for an ApsaraDB for MongoDB instance. For more information, see Overview of data migration scenarios.
What is the difference between a data migration task whose source instance is an RDS instance and a data migration task whose source instance is a self-managed database instance that is connected over a public IP address?
If you select an RDS instance as the source instance, the data migration task can be adaptive to modifications, such as the modifications of Domain Name System (DNS) and network type changes. This ensures instance reliability.
Am I able to migrate data from a self-managed database that is hosted on an ECS instance in a VPC to an RDS instance?
Yes, you can migrate a self-managed database hosted on an ECS instance in a VPC to an RDS instance.
If the ECS instance and the destination RDS instance reside in the same region, DTS can directly access the self-managed database on the ECS instance in the VPC.
If the ECS instance and the destination RDS instance reside in different regions, the ECS instance needs to be associated with an elastic IP address (EIP). In this case, DTS automatically uses the EIP of the ECS instance to access the database hosted on the ECS instance.
Are tables locked by DTS during data migration? Does it affect the source database?
DTS does not lock tables in the source database during full data migration or incremental data migration. During full data migration and incremental data migration, tables in the source database can be read and written.
Does DTS obtain data from the primary or secondary RDS database when DTS migrates data from an RDS instance?
DTS pulls data from the primary RDS database when DTS migrates data from an RDS instance.
Am I able to configure a data migration task that automatically starts at the specified point in time?
No, you cannot configure a data migration task that automatically starts at the specified point in time.
Am I able to migrate data from an RDS instance in a VPC?
Yes, you can migrate data from an RDS instance in a VPC. Specify the ID of the RDS instance when you configure a data migration task.
During data migration or synchronization within the same Alibaba Cloud account or across Alibaba Cloud accounts, does DTS access an ECS instance or RDS instance over an internal network or the Internet? Am I charged for network traffic?
Whether data migration or synchronization is performed within the same Alibaba Cloud account or across Alibaba Cloud accounts does not affect the network type. Whether you are charged for network traffic is determined by the DTS instance type.
Network type
Data migration: If you migrate data within the same region, DTS accesses the ECS instance or RDS instance over an internal network. If you migrate data across regions, DTS accesses the source ECS instance or RDS instance over the Internet and accesses the destination RDS instance over an internal network.
Data synchronization: DTS accesses the ECS instance or RDS instance over an internal network.
Traffic fees
Data migration: You are charged for outbound traffic over the Internet. You are not charged for traffic over the Internet for other types of DTS instances. Fees for outbound traffic over the Internet are generated only when the Access Method parameter of the destination instance is set to Public IP Address.
Data synchronization: You are not charged for traffic over the Internet.
Is the data in the source database deleted after the data is migrated by using DTS?
No, the data in the source database is not deleted. During the migration, DTS copies the data in the source database to the destination database. This does not affect the data in the source database.
Am I able to specify the name of the destination database when I use DTS to migrate data between RDS instances?
Yes, you can skip full data migration. During the migration, you can use the database name mapping feature that is provided by DTS to specify the name of the destination database. For more information, see Synchronize or migrate data between databases with different names.
What do I do if DTS cannot connect to the source ECS instance in a data migration task?
The public IP address of the ECS instance may be unavailable. Associate an elastic IP address (EIP) with the ECS instance and try again. For more information about how to associate an EIP, see Associate or disassociate an EIP.
Why is Full Data Synchronization not displayed when DTS migrates data between Redis instances?
Incremental Data Migration is displayed. If you select this option for data migration between Redis instances, both full data migration and incremental data migration are performed.
Can I skip full data migration?
Yes, you can skip full data migration. After you skip full data migration, the incremental data migration continues. However, an error may occur. We recommend that you do not skip full data migration.
Am I able to connect a Redis instance of the cluster edition to DTS over a public IP address?
No, you can only connect a Redis instance of the standalone edition to DTS over a public IP address.
What items do I need to take note of when I migrate data from a MySQL 8.0 instance to a MySQL 5.6 instance?
You must create a database in the MySQL 5.6 instance before you start the data migration task. To ensure compatibility, make sure that the version of the destination database is the same as or later than that of the source database. If the version of the destination database is earlier than the version of the source database, database compatibility issues may occur.
Am I able to migrate accounts from the source database to the destination database?
You can migrate accounts only between ApsaraDB RDS for MySQL instances.
FAQ about change tracking
How does DTS work in change tracking mode?
For more information about how DTS works in change tracking mode, see Architecture.
Are consumer groups deleted after a change tracking instance expires?
After a change tracking instance expires, relevant consumer groups are retained for seven days. If the instance is not renewed within seven days after expiration, it is released, and relevant consumer groups are also deleted.
Can a read-only database instance be used as the source instance for a change tracking task?
Take note of the following items:
If the read-only database instance records transaction logs, such as an instance of ApsaraDB RDS for MySQL 5.7 or 8.0, it can be used as the source instance.
If the read-only database instance does not record transaction logs, such as an instance of ApsaraDB RDS for MySQL 5.6, it cannot be used as the source instance.
How do I consume tracked data?
For more information about how to consume tracked data, see Consume tracked data.
Why is the date format changed after data is transmitted by using the change tracking feature?
By default, a date is stored in the YYYY:MM:DD format but displayed in the YYYY-MM-DD format in DTS. Therefore, all dates are stored and displayed in the default formats in DTS regardless of the original formats in which the dates are transmitted and written.
What do I do if the SDK client is suddenly suspended during data download and cannot track data changes?
Check whether the ackAsConsumed method is called in the SDK code to report a consumer offset. If the ackAsConsumed method is not called to report a consumer offset, data in the cache cannot be cleared. The SDK fails to pull data after the cache is fully occupied. In this case, the SDK is suspended and cannot track data changes.
What do I do if I cannot track data changes after the SDK client is restarted?
Before you start the SDK client, you need to modify the consumer offset to be within the time range of all tracked data changes. For more information, see the "Manage consumer offsets" section of the Use the SDK demo to consume tracked data topic.
How do I specify a consumer offset for a client?
You can configure the initCheckpoint parameter to specify a consumer offset for the client. For more information, see Use the SDK demo to consume tracked data.
How do I reset the consumer offset if the number of change tracking tasks exceeds the limit?
Open the corresponding code file, such as the DTSConsumerAssignDemo.java or DTSConsumerSubscribeDemo.java file, based on the mode in which you use the SDK client.
In the Data Range column on the Change Tracking page, check the data range within which you can reset the consumer offset in the change tracking instance.
Select a new consumer offset based on your business requirements and convert the consumer offset to a UNIX timestamp.
Specify the initCheckpoint parameter in the code file to replace the original consumer offset with the new UNIX timestamp.
Restart the SDK client.
What do I do if I cannot access a change tracking task over VPC by using a client?
This may happen if the server on which the client is hosted is not in the VPC that you specified when you configured the change tracking task. For example, the VPC of the client is changed. In this case, you must reconfigure the change tracking task.
Why is the value of the consumer offset in the console greater than the maximum value in the data range?
The data range of the change tracking task is updated every minute, whereas the consumer offset is updated every 10 seconds. Therefore, if you consume data in real time, the value of the consumer offset may be greater than the maximum value in the data range of the change tracking task.
How does DTS ensure that the data that is tracked by using SDK is a complete transaction?
Based on the provided consumer offset, DTS searches for the complete transaction that corresponds to the consumer offset and distributes data to downstream from the BEGIN statement of the entire transaction. Therefore, a complete transaction can be received.
How do I check whether the tracked data is properly consumed?
If the data is properly consumed, the consumer offset that is displayed in the DTS console moves forward.
What does usePublicIp=true indicate in the change tracking SDK code?
If you set the usePublicIp parameter to true, the SDK accesses the DTS change tracking task over the Internet.
Are my services affected if the primary and secondary databases of the source RDS instance that is configured for the change tracking task are switched over or the primary database is restarted?
No, your services are not affected. DTS adapts to the changes if the primary and secondary databases are switched over or the primary database is restarted in an ApsaraDB RDS for MySQL instance, an ApsaraDB RDS for PostgreSQL instance, a PolarDB for MySQL instance, a PolarDB for PostgreSQL instance, or a PolarDB-X 1.0 instance whose storage type is ApsaraDB RDS for MySQL.
Can RDS automatically download binary logs to a local server?
DTS allows you to track RDS binary logs in real time. You can enable the change tracking feature of DTS to track RDS binary logs and synchronize the logs to your local server in real time by using DTS SDK.
Does the real-time incremental data of change tracking include only new data or both new data and modified data?
The incremental data that can be tracked by DTS includes all added, deleted, and modified data and the changes of schemas caused by DDL operations.
Why do I receive some messages multiple times after the SDK client is restarted if a message is not acknowledged at a consumer client?
If the SDK contains a message that is not acknowledged, the server pushes all messages in the buffer. After the messages are pushed, the SDK can no longer receive messages. In this case, the consumer offset that is stored on the server is the checkpoint of the last message before the message that is not acknowledged. When the SDK client restarts, the server pushes data from the checkpoint of the last message before the message that is not acknowledged to ensure that no messages are lost. Therefore, the SDK client receives some messages multiple times at this time.
How often is the consumer offset of change tracking updated? Why do I sometimes receive a message multiple times when I restart the SDK client?
After the change tracking SDK consumes each message, the SDK must call the ackAsConsumed method to send an acknowledgment to the server. After the server receives the acknowledgment, the server updates the consumer offset in the memory and persists the consumer offset every 10 seconds. If you restart the SDK client before the latest acknowledgment is persisted, the server pushes messages from the consumer offset of the message whose acknowledgment is most recently persisted to make sure that no messages are lost. In this case, the SDK receives a message multiple times.
Am I able to track data in multiple RDS instances by using a change tracking instance?
No, you cannot track data in multiple RDS instances by using a change tracking instance. A change tracking instance can only track data in a single RDS instance.
Does data inconsistency occur in a change tracking instance?
No, change tracking tasks only obtain the changes of the source database and do not involve data inconsistency. If the data that is consumed by the consumer client does not meet expectations, troubleshoot the issue.
What do I do if the "UserRecordGenerator: haven't receive records from generator for 5s" error message appears when I consume tracked data?
If the UserRecordGenerator: haven't receive records from generator for 5s error message appears when you consume tracked data, make sure that the consumer offset is within the offset range of the incremental data collection module and that the consumer client runs as expected.
Am I able to create multiple partitions for a topic?
No, you cannot create multiple partitions for a topic. To ensure the global order of messages, DTS allows you to create only one partition for each topic. The topic is allocated to partition 0.
What is the impact of modifying the data of the destination database when a data synchronization or migration task is in progress?
The data synchronization or migration task may fail. If you perform operations on the objects in the destination database during data synchronization or migration, exceptions may occur. For example, a primary key conflict may occur, or data records are not updated. In this case, the data synchronization or migration task may fail. If you perform operations that do not interrupt data synchronization or migration, the task is not affected. For example, if you create a table in the destination database and write data to the table, the task is not affected because the table is not an object to be migrated or synchronized.
DTS reads the information about the source database and migrates or synchronizes full data, schema data, and incremental data to the destination database. Therefore, modifications of data in the destination database may be overwritten by migrated or synchronized data.
What happens if I change the password of the source or destination database when a DTS instance is in use?
DTS returns an error and stops the instance. You can log on to the DTS console and click the instance ID to go to the instance details page. On the Basic Information page, you can change the account and password of the source or destination database. Then, go to the Task Management page. On the Task Management page, find the module for which an error is reported and restart the module.
Why am I unable to connect to some source or destination databases over a public IP address?
The connection type varies based on the task type and the type of the source or destination database. For example, if you select a MySQL database as the source database, you can connect to the source database over a public IP address for a data migration or change tracking task, but not for a data synchronization task.
The following error is reported for a DTS task whose destination database resides in a Redis instance: OOM command not allowed when used memory > 'maxmemory'. Why does this happen?
This error may be caused due to insufficient storage space of the destination Redis instance. If the destination Redis instance uses the cluster architecture, this error may be caused because the memory usage of a shard reaches the upper limit. In this case, upgrade the specifications of the Redis instance.
What is the AliyunDTSRolePolicy policy?
The AliyunDTSRolePolicy policy is used to access cloud resources such as ApsaraDB RDS instances and ECS instances within your Alibaba Cloud account or across Alibaba Cloud accounts when you configure DTS tasks. For more information, see Authorize DTS to access Alibaba Cloud resources.
How do I grant permissions to a RAM role?
When you log on to the DTS console for the first time, you are required to grant permissions to the AliyunDTSDefaultRole role. Grant permissions to the RAM role in the console as prompted. For more information, see Authorize DTS to access Alibaba Cloud resources.
Important
You must log on to the DTS console by using your Alibaba Cloud account.
Am I able to change the password of a database account for a DTS task?
Yes, you can change the password of a database account for a DTS task. You can log on to the DTS console and click the instance ID to go to the instance details page. On the Basic Information page, click Change Password to change the password of the source or destination database account.
Important
You cannot change the password of the system account for a DTS task.
Why do MaxCompute table names end with the _base suffix?
Initial schema synchronization.
DTS synchronizes the schemas of the required objects from the source database to MaxCompute. During initial schema synchronization, DTS adds the _base suffix to the end of the source table name. For example, if the name of the source table is customer, the name of the table in MaxCompute is customer_base.
Initial full data synchronization.
DTS synchronizes the historical data of the table from the source database to the destination table in MaxCompute. For example, the customer table in the source database is synchronized to the customer_base table in MaxCompute. The data is the basis for subsequent incremental synchronization.
Note
The destination table that is suffixed with _base is known as a full baseline table.
Incremental data synchronization.
DTS creates an incremental data table in MaxCompute. The name of the incremental data table is suffixed with _log, such as customer_log. Then, DTS synchronizes the incremental data that was generated in the source database to the incremental data table.
What do I do if I cannot obtain the information about Kafka topics?
This may be because no topic information exists on Kafka brokers. You can run the following command to check the distribution of topics among Kafka brokers:
./bin/kafka-topics.sh --describe --zookeeper zk01:2181/kafka --topic topic_name
Am I able to create a local MySQL instance as the secondary database of an RDS instance?
Yes, you can use the data migration feature of DTS to migrate data from the RDS instance to the local MySQL instance in real time. Then, you can use the MySQL instance as the secondary database of the RDS instance.
How do I replicate data from an RDS instance to a newly created RDS instance?
You can use the data migration feature of DTS. When you configure a data migration task, select schema migration, full data migration, and incremental data migration as the migration types. For more information, see Migrate data between ApsaraDB RDS instances.
Am I able to replicate a database that is the same as another database except for the database name in an RDS instance?
Yes, DTS allows you to use the object name mapping feature to replicate a database that is the same as another database except for the database name in an RDS instance.
What do I do if latency always occurs in a DTS instance?
Latency may always occur in a DTS instance due to the following causes:
You created multiple DTS tasks in the source instance by using different accounts. As a result, the source instance is heavily loaded. We recommend that you create DTS tasks by using the same account.
The memory of the destination instance may be insufficient. Arrange your business and restart the destination instance. If the issue persists, upgrade the destination instance or perform a primary-secondary switchover.
Note
During the switchover, the network connection may be interrupted. Make sure that your application can automatically reconnect to the instance.
What do I do if the fields are in lowercase after I synchronize or migrate data to the destination database in the DTS console of the previous version?
Configure tasks in the DTS console of the new version and specify the capitalization of object names in the destination database. For more information, see Specify the capitalization of object names in the destination instance.
Am I able to resume a DTS task after the task is paused?
In most cases, a DTS task that is paused for no more than 24 hours can be properly resumed. If the amount of data is small, a DTS task that is paused for no more than seven days can be resumed. We recommend that you do not pause a task for more than 6 hours.
Why does the progress of a task start from 0 after the task is paused and then restarted?
After the task is restarted, DTS queries the data that is transmitted again and continues to process the remaining data. During this process, the task progress may differ from the actual situation due to latency.
What is the principle of lock-free DDL?
For more information about the principle of lock-free DDL operations, see the "Lock-free schema change process" section of the Overview topic.
Am I able to pause the data synchronization or migration of a table?
No, you cannot pause the data synchronization or migration of a table.
If a task fails, do I need to purchase the task again?
No, you can reconfigure the task.
What happens if multiple tasks write data to the same destination database?
This may cause data inconsistency.
Why is a DTS instance still locked after the instance is renewed?
After you renew a locked DTS instance, it takes a while for the instance to be unlocked.
Am I able to modify the resource group of a DTS instance?
Yes, you can modify the resource group of a DTS instance. You can go to the Basic Information page of the instance. In the Basic Information section, click Edit next to the resource group name.
Does DTS provide a binary log analysis tool?
No, DTS does not provide a binary log analysis tool.
Is it normal that the progress of an incremental data synchronization or migration task is always displayed as 95%?
Yes, it is normal. Incremental data synchronization or migration tasks never stop or complete. The progress cannot reach 100%.
Why is a DTS task not released after it is frozen for more than seven days?
In some cases, frozen tasks are saved for more than seven days.
Am I able to modify the port number of an existing task?
No, you cannot modify the port number of an existing task.
Am I able to downgrade an ApsaraDB RDS for MySQL instance that is mounted to a PolarDB-X 1.0 instance in a DTS task?
We recommend that you do not downgrade the ApsaraDB RDS for MySQL instance. The downgrade operation triggers a primary-secondary switchover, which may cause data loss.
Am I able to upgrade or downgrade the source or destination instance when a DTS task is in progress?
If you upgrade or downgrade the source or destination instance when a DTS task is in progress, the task may be delayed or the data may be lost. We recommend that you do not modify the specifications of the source or destination instance when a DTS task is in progress.
How do DTS tasks affect the source and destination instances?
During full data initialization, DTS uses the read and write resources of the source and destination databases. This may increase the load on the databases. We recommend that you run full data migration or synchronization tasks during off-peak hours.
What is the approximate latency of a DTS task?
The latency of DTS tasks cannot be estimated because the latency is affected by various factors, such as the running load of the source instance, the bandwidth of the transmission network, the network latency, and the write performance of the destination instance.
How do I return to the DTS console of the previous version if I am navigated to the DMS console?
You can move the pointer over the 
image in the lower-right corner of the DMS console and click the 
icon to go to the DTS console of the previous version.
Does DTS support data encryption?
Yes, DTS supports secure database access via SSL when reading data from the source database and writing into the target database. However, the data transfer encryption between the source and target is NOT supported.
Am I able to configure an ApsaraDB for ClickHouse instance as the source or destination instance?
No, you cannot configure an ApsaraDB for ClickHouse instance as the source or destination instance.
Am I able to configure an AnalyticDB for MySQL V2.0 cluster as the source or destination instance?
You can configure an AnalyticDB for MySQL V2.0 cluster only as a destination instance. You can perform the operation only in the DTS console of the previous version, but not in the DTS console of the new version.AnalyticDB for MySQL
Why am I unable to view a newly created task in the DTS console?
You may select an incorrect resource group or configure filter conditions for tasks. Configure correct filter conditions for the tasks. For example, select the correct region and resource group in the top navigation bar.
Why does data inconsistency occur for a data verification task?
Data inconsistency may occur for a data verification task due to the following causes:
Latency occurs in a data migration or synchronization task.
Columns with default values are added to the source database and latency occurs in the task.
You use tools other than DTS to write data to the destination database.
DDL operations are performed on the source database of the task for which the multi-table merging feature is enabled.
The object name mapping feature is enabled for the data migration or data synchronization task.
Am I able to modify the gray configuration items of a created task?
No, you cannot modify the gray configuration items of a created task.
How can I configure latency alerts and thresholds?
DTS provides the monitoring and alerting feature for DTS tasks. You can configure alert rules for important metrics in the DTS console to monitor DTS tasks in real time. For more information, see Configure monitoring and alerting.
Am I able to view the cause of a task failure that happened an extended period of time ago?
No, you cannot view the cause of a task failure that happened an extended period of time ago? If a task failed an extended period of time ago, such as more than seven days ago, related logs are cleared. In this case, you cannot view the cause of the failure.
Am I able to resume a task that failed an extended period of time ago?
No, you cannot resume a task that failed an extended period of time ago. If a task failed an extended period of time ago, such as more than seven days ago, related logs are cleared. In this case, you must reconfigure the task.
What is the rdsdt_dtsacct account?
If you do not create the rdsdt_dtsacct account, the account may be created by DTS. DTS creates the built-in database account named rdsdt_dtsacct in some database instances to connect to the source and destination database instances.
How do I view the information about heap tables, tables without primary keys, compressed tables, and tables that contain computed columns in an SQL Server database?
You can execute the following SQL statements to check whether the source database contains these types of tables:
Execute the following SQL statement to check for heap tables:
SELECT s.name AS schema_name, t.name AS table_name FROM sys.schemas s INNER JOIN sys.tables t ON s.schema_id = t.schema_id AND t.type = 'U' AND s.name NOT IN ('cdc', 'sys') AND t.name NOT IN ('systranschemas') AND t.object_id IN (SELECT object_id FROM sys.indexes WHERE index_id = 0);
Execute the following SQL statement to check for tables without primary keys:
SELECT s.name AS schema_name, t.name AS table_name FROM sys.schemas s INNER JOIN sys.tables t ON s.schema_id = t.schema_id AND t.type = 'U' AND s.name NOT IN ('cdc', 'sys') AND t.name NOT IN ('systranschemas') AND t.object_id NOT IN (SELECT parent_object_id FROM sys.objects WHERE type = 'PK');
Execute the following SQL statement to check for primary key columns that are not contained in clustered index columns:
SELECT s.name schema_name, t.name table_name FROM sys.schemas s INNER JOIN sys.tables t ON s.schema_id = t.schema_id WHERE t.type = 'U' AND s.name NOT IN('cdc', 'sys') AND t.name NOT IN('systranschemas') AND t.object_id IN ( SELECT pk_colums_counter.object_id AS object_id FROM (select pk_colums.object_id, sum(pk_colums.column_id) column_id_counter from (select sic.object_id object_id, sic.column_id FROM sys.index_columns sic, sys.indexes sis WHERE sic.object_id = sis.object_id AND sic.index_id = sis.index_id AND sis.is_primary_key = 'true') pk_colums group by object_id) pk_colums_counter inner JOIN ( select cluster_colums.object_id, sum(cluster_colums.column_id) column_id_counter from (SELECT sic.object_id object_id, sic.column_id FROM sys.index_columns sic, sys.indexes sis WHERE sic.object_id = sis.object_id AND sic.index_id = sis.index_id AND sis.index_id = 1) cluster_colums group by object_id ) cluster_colums_counter ON pk_colums_counter.object_id = cluster_colums_counter.object_id and pk_colums_counter.column_id_counter != cluster_colums_counter.column_id_counter);
Execute the following SQL statement to check for compressed tables:
SELECT s.name AS schema_name, t.name AS table_name FROM sys.objects t, sys.schemas s, sys.partitions p WHERE s.schema_id = t.schema_id AND t.type = 'U' AND s.name NOT IN ('cdc', 'sys') AND t.name NOT IN ('systranschemas') AND t.object_id = p.object_id AND p.data_compression != 0;
Execute the following SQL statement to check for tables with computed columns:
SELECT s.name AS schema_name, t.name AS table_name FROM sys.schemas s INNER JOIN sys.tables t ON s.schema_id = t.schema_id AND t.type = 'U' AND s.name NOT IN ('cdc', 'sys') AND t.name NOT IN ('systranschemas') AND t.object_id IN (SELECT object_id FROM sys.columns WHERE is_computed = 1);
What do I do if the schemas of the source and destination databases are inconsistent?
You can use the object name mapping feature to map the columns of the source database to those of the destination database. For more information, see Map object names.
Note
You cannot modify the type of a column.
Am I able to modify the type of a column by using the object name mapping feature?
No, you cannot modify the type of a column by using the object name mapping feature.
Am I able to limit the speed at which the source database is read?
No, you cannot limit the speed at which the source database is read. Before you start a task, you must evaluate the performance of the source database, such as whether the IOPS and network bandwidth meet the requirements of the source database. We recommend that you run the task during off-peak hours.
How do I delete orphaned documents of a MongoDB database deployed in the sharded cluster architecture?
Check whether orphaned documents exist
Connect to the MongoDB database deployed in the sharded cluster architecture by using the mongo shell.
For more information about how to connect to an ApsaraDB for MongoDB sharded cluster instance, see Connect to a sharded cluster instance by using the mongo shell.
Run the following command to switch to the destination MongoDB database:
Run the following command to view the information about orphaned documents:
db.<coll_name>.find().explain("executionStats")
Note
View the value of the chunkSkips field of the SHARDING_FILTER stage in executionStats of each shard. If the value is not 0, orphaned documents exist in the corresponding shard.
The following response indicates that 102 documents are returned in the FETCH stage. Then, 2 orphaned documents are filtered out in the SHARDING_FILTER stage and 100 documents are returned.
"stage" : "SHARDING_FILTER",
"nReturned" : 100,
......
"chunkSkips" : 2,
"inputStage" : {
"stage" : "FETCH",
"nReturned" : 102,
For more information about the SHARDING_FILTER stage, see MongoDB Manual.
Delete orphaned documents
Important
If you have multiple MongoDB databases, you must delete orphaned documents from each MongoDB database.
ApsaraDB for MongoDB instances
Self-managed MongoDB databases
The cleanupOrphaned command is required to delete orphaned documents. The method of running this command varies based on the version of the MongoDB database.
MongoDB 4.4 and later
MongoDB 4.2 and earlier
Create a JavaScript script file named cleanupOrphaned.js on a server that can connect to the sharded cluster instance.
Note
This script is used to delete orphaned documents from all collections in multiple databases in multiple shards. If you want to delete orphaned documents from a specific collection, you can modify some of the parameters in the script file.
// The names of shards.
var shardNames = ["shardName1", "shardName2"];
// The databases from which you want to delete orphaned documents.
var databasesToProcess = ["database1", "database2", "database3"];
shardNames.forEach(function(shardName) {
// Traverse the specified databases.
databasesToProcess.forEach(function(dbName) {
var dbInstance = db.getSiblingDB(dbName);
// Obtain the names of all collections of the specified databases.
var collectionNames = dbInstance.getCollectionNames();
// Traverse all collections.
collectionNames.forEach(function(collectionName) {
// The complete collection name.
var fullCollectionName = dbName + "." + collectionName;
// Build the cleanupOrphaned command.
var command = {
runCommandOnShard: shardName,
command: { cleanupOrphaned: fullCollectionName }
};
// Run the cleanupOrphaned command.
var result = db.adminCommand(command);
if (result.ok) {
print("Cleaned up orphaned documents for collection " + fullCollectionName + " on shard " + shardName);
printjson(result);
} else {
print("Failed to clean up orphaned documents for collection " + fullCollectionName + " on shard " + shardName);
}
});
});
});
You must modify the shardNames and databasesToProcess parameters in the script file. The following content describes the two parameters:
shardNames: the IDs of the shards from which you want to delete orphaned documents. You can view the IDs in the Shard List section on the Basic Information page of the sharded cluster instance. Example: d-bp15a3796d3a****.
databasesToProcess: the names of the databases from which you want to delete orphaned documents.
Run the following command in the directory in which the cleanupOrphaned.js script file is stored:
mongo --host <Mongoshost> --port <Primaryport> --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js > output.txt
The following table describes the parameters that you can configure.
Parameter | Description |
<Mongoshost>
| The endpoint of the mongos node of the sharded cluster instance. Format: s-bp14423a2a51****.mongodb.rds.aliyuncs.com. |
<Primaryport>
| The port number of the mongos node of the sharded cluster instance. Default value: 3717. |
<database>
| The name of the database to which the database account belongs. |
<username>
| The database account. |
<password>
| The password of the database account. |
output.txt
| The output.txt file that is used to store execution results. |
Create a JavaScript script file named cleanupOrphaned.js on a server that can connect to the sharded cluster instance.
Note
This script is used to delete orphaned documents from a specific collection in a database in multiple shards. If you want to delete orphaned documents from multiple collections, you can modify the fullCollectionName parameter in the script file and run the script multiple times. Alternatively, you can modify the script file to traverse all collections.
function cleanupOrphanedOnShard(shardName, fullCollectionName) {
var nextKey = { };
var result;
while ( nextKey != null ) {
var command = {
runCommandOnShard: shardName,
command: { cleanupOrphaned: fullCollectionName, startingFromKey: nextKey }
};
result = db.adminCommand(command);
printjson(result);
if (result.ok != 1 || !(result.results.hasOwnProperty(shardName)) || result.results[shardName].ok != 1 ) {
print("Unable to complete at this time: failure or timeout.")
break
}
nextKey = result.results[shardName].stoppedAtKey;
}
print("cleanupOrphaned done for coll: " + fullCollectionName + " on shard: " + shardName)
}
var shardNames = ["shardName1", "shardName2", "shardName3"]
var fullCollectionName = "database.collection"
shardNames.forEach(function(shardName) {
cleanupOrphanedOnShard(shardName, fullCollectionName);
});
You must modify the shardNames and fullCollectionName parameters in the script file. The following content describes the two parameters:
shardNames: the IDs of the shards from which you want to delete orphaned documents. You can view the IDs in the Shard List section on the Basic Information page of the sharded cluster instance. Example: d-bp15a3796d3a****.
fullCollectionName: You must replace this parameter with the name of the collection from which you want to delete orphaned documents. Format: database name.collection name.
Run the following command in the directory in which the cleanupOrphaned.js script file is stored:
mongo --host <Mongoshost> --port <Primaryport> --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js > output.txt
The following table describes the parameters that you can configure.
Parameter | Description |
<Mongoshost>
| The endpoint of the mongos node of the sharded cluster instance. Format: s-bp14423a2a51****.mongodb.rds.aliyuncs.com. |
<Primaryport>
| The port number of the mongos node of the sharded cluster instance. Default value: 3717. |
<database>
| The name of the database to which the database account belongs. |
<username>
| The database account. |
<password>
| The password of the database account. |
output.txt
| The output.txt file that is used to store execution results. |
Download the cleanupOrphaned.js script file on a server that can connect to the self-managed MongoDB database.
wget "https://docs-aliyun.cn-hangzhou.oss.aliyun-inc.com/assets/attach/120562/cn_zh/1564451237979/cleanupOrphaned.js"
Replace test in the cleanupOrphaned.js file with the name of the database from which you want to delete orphaned documents.
Important
If you want to delete orphaned documents from multiple databases, repeat Step 2 and Step 3.
Run the following command on a shard to delete the orphaned documents from all collections in the specified database:
Note
You must repeat this step for each shard.
mongo --host <Shardhost> --port <Primaryport> --authenticationDatabase <database> -u <username> -p <password> cleanupOrphaned.js
Note
<Shardhost>: the IP address of the shard.
<Primaryport>: the service port of the primary node in the shard.
<database>: the name of the database to which the database account belongs.
<username>: the account that is used to log on to the self-managed MongoDB database.
<password>: the password that is used to log on to the self-managed MongoDB database.
Example:
In this example, a self-managed MongoDB database has three shards, and you must delete the orphaned documents from each shard.
mongo --host 172.16.1.10 --port 27018 --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js
mongo --host 172.16.1.11 --port 27021 --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js
mongo --host 172.16.1.12 --port 27024 --authenticationDatabase admin -u dtstest -p 'Test123456' cleanupOrphaned.js
Troubleshooting
If idle cursors exist in the namespace of orphaned documents, the deletion may fail and the following mongod log is recorded for the corresponding orphaned documents.
Deletion of DATABASE.COLLECTION range [{ KEY: VALUE1 }, { KEY: VALUE2 }) will be scheduled after all possibly dependent queries finish
You can use the mongo shell to connect to the mongod process and run the following command to check whether idle cursors exist in the current shard. If idle cursors exist, run the restart mongod or killCursors command to clear all idle cursors. Then, delete orphaned documents again. For more information, see Open cursors can block cleanupOrphaned.
db.getSiblingDB("admin").aggregate( [{ $currentOp : { allUsers: true, idleCursors: true } },{ $match : { type: "idleCursor" } }] )
What do I do if the data of a MongoDB database deployed in the sharded cluster architecture is not evenly distributed?
You can enable the balancer and perform pre-sharding to effectively resolve the issue that most data is written to a single shard.
Enable the balancer
If the balancer is disabled or the specified active time window of the balancer is not reached, you can enable the balancer or temporarily cancel the time window of the balancer. This way, you can immediately balance the data distribution.
Connect to the MongoDB database deployed in the sharded cluster architecture.
After you connect to a mongos node, run the following command in the mongo shell to switch to the config database:
Perform the following operations based on your business requirements.
Perform pre-sharding
MongoDB allows you to configure ranged sharding or hash sharding. You can perform pre-sharding to distribute the values of chunks to as many shard nodes as possible. This way, the data can be evenly distributed during data synchronization or migration.
Perform hash sharding
Perform ranged sharding
You can specify the numInitialChunks parameter to perform pre-sharding in a more efficient manner. The default value is the result of the number of shards multiplied by two. The maximum value is the result of the number of shards multiplied by 8,192. For more information, see sh.shardCollection().
sh.shardCollection("phonebook.contacts", { last_name: "hashed" }, false, {numInitialChunks: 16384})
If the source database is a MongoDB sharded cluster instance, you can use the data in config.chunks to obtain the chunk range of the sharded tables in the source MongoDB instance. The chunk range can be used as a reference to determine the values of <split_value> in subsequent pre-sharding.
If the source database is a MongoDB replica set instance, you must run the find command to specify the value range of the sharding key. Then, you can specify a proper shard point.
# Obtain the minimum value of the sharding key.
db.<coll>.find().sort({<shardKey>:1}).limit(1)
# Obtain the maximum value of the sharding key.
db.<coll>.find().sort({<shardKey>:-1).limit(1)
Command syntax
sh.splitAt("<db>.<coll>", {"<shardKey>":<split_value>})
Examples
sh.splitAt("test.test", {"id":0})
sh.splitAt("test.test", {"id":50000})
sh.splitAt("test.test", {"id":75000})
After the pre-sharding is performed, you can run the sh.status() command on the mongos node to check the pre-sharding result.
Elastic Compute Service (ECS)
Container Compute Service (ACS)
