Data Transmission Service (DTS): Migrate Data Between Different Data Storage Types

The Definition of Data Transmission Services

Alibaba Cloud Data Transmission Service (DTS) helps you migrate data between data stores, such as relational databases, NoSQL databases, and data warehouses. You can use DTS to migrate your data to Alibaba Cloud or between combinations of cloud and on-premises data systems.

Data Transmission Service (DTS) supports several data replication modes, including data migration, data integration, data synchronization, and change tracking. You can choose a combination of data replication modes that best suit your use cases.

The Features of Data Transmission Services

As a managed service, Data Transmission Service (DTS) offers the following advantages over traditional data replication tools:

Provides highly stable data transmission.
Helps you manage data migration between your data stores so that you can focus on developing applications.
Provides several data replication modes, including data migration, data integration, data synchronization, and change tracking.
Supports data migration between data stores that are based on different engines and architectures.
Supports a variety of database engines as sources and targets.
Supports resumable transmission that addresses transmission interruptions caused by hardware and network failures.
Helps you build a distributed data system that is scalable and highly available.
Supports RAM-based authorization that allows you to define fine-grained authorization policies for managing DTS tasks.
Supports scheduled migration tasks for handling recurring data migration workloads.

The Architecture of Data Transmission Service (DTS)

The architecture of Data Transmission Services provides the following features:

Primary/secondary redundancy:
Each function of DTS is deployed on multiple servers with primary/secondary redundancy. The HA manager continuously performs health checks on each server. If one server functions abnormally, the workloads on that server are switched to a healthy server with minimal delay.
Endpoint change detection:
For continuous data replications, such as data synchronization and change tracking, the HA manager detects the changes made to the endpoints of source ApsaraDB instances. If an instance endpoint has changed, the HA manager reconfigures the data source to keep the data connection functioning.

Architecture_of_Data_Transmission_Service

How Data Transmission Service (DTS) Works in Data Migration Mode

A data migration task consists of several phases, namely schema migration, full data migration, and incremental data migration. To keep the source data operational during the migration, you must select all of these phases in the migration task configuration wizard.

Before migrating data, DTS needs to re-create the schema in the target database. For heterogeneous migrations, DTS parses the data definition language (DDL) code of the source database, translates the code into the syntax of the target database, and then re-creates the schema objects in the target database.

In the full data migration phase, DTS replicates the existing data from the source database to the target database. The source database remains operational and updates are continuously made during the migration process. DTS uses an incremental data reader to capture the ongoing changes that occur during the full data migration process. The incremental data reading is activated when the full data migration starts. During the full data migration phase, incremental data is parsed, reformatted, and stored locally on the DTS server.

When the full data migration process is complete, DTS retrieves the incremental data stored locally, reformats it again, and applies the incremental changes in the target database. This process continues until all ongoing updates are replicated to the target database and the source and target databases are in sync.

Data_migration_process

How Data Transmission Service (DTS) Works in Data Synchronization Mode

The data synchronization mode of DTS replicates ongoing changes between two data stores. This mode is typically used for OLTP-to-OLAP replications. In this mode, a migration task consists of the following two phases:

Initial data load: DTS loads the existing data from the source database to the target database.
Ongoing replication: DTS replicates ongoing changes and keeps the source and target databases in sync.

To replicate ongoing changes, DTS uses two components that work with the transaction log:

Transaction log reader: The transaction log reader communicates with the source database using the corresponding protocol to read the transaction log. For example, it uses Binglog Dump to read transaction log data from ApsaraDB for MySQL databases.
Transaction log applier: The transaction log applier retrieves data updates from the transaction log reader, filters the updates to keep only ones related to the objects being replicated, and applies the updates to the target database. When doing this, the transaction log applier maintains the ACID properties of transactions. Both the transaction log reader and the transaction log applier are based on redundant deployments. The HA manager checks the health condition of each server. If an anomaly occurs, the execution of the transaction log is resumed on a healthy server.

Data_synchronization_process_3

How Data Transmission Service (DTS) Works in Change Tracking Mode

The change tracking replication mode of DTS captures data updates and exposes them as a publisher/subscriber stream. You can customize the consumption mechanism for your different applications.

The log processor communicates with the source database using the corresponding protocol to read the transaction log. For example, it uses Binglog Dump to read transaction log data from ApsaraDB for MySQL databases. Then, the log processor parses the transaction log data, filters the data, normalizes the update records, and keeps the processed data in persistence.

The log processor is based on redundant deployments. The HA manager checks the health condition of each server. If an anomaly occurs, the workloads of transaction log reading are resumed on a healthy server.
Change_tracking_process_4

Related Products