Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Getting Started with Data Transmission Service

Getting Started with Data Transmission Service

This article explains what the Data Transmission service is and how to synchronize data from ApsaraDB RDS for MySQL to Alibaba Cloud Elasticsearch.

Data Transmission Service – What and How | Alibaba Cloud

This article explains what the Data Transmission service is and how you can use it to the fullest.

Data Transmission Service (DTS) is a fully managed data replication service that supports a variety of operations and modes. Migrations between relational databases, data warehouses, and NoSQL databases are all supported. The Data Transmission Service (DTS) allows you to transfer data incrementally from on-premises to the cloud and from cloud to cloud.
Data Transmission Service

Data Transmission Service (DTS) supports multiple replication modes to replicate data:

  1. Data Migration: This mode is used to reduce any downtime since it is mostly used for one-time migration scenarios. Incremental data migration can be used to replicate changes in real-time.
  2. Data Integration: This mode is used for larger migration jobs. In this mode, DTS schedules the migration tasks and migrates the data at regular intervals.
  3. Data Synchronization: As the name suggests, this mode is used to synchronize data in real-time between two data stores. This is recommended most for a distributed system.
  4. Change Tracking: This mode is used to capture changes made to the data store.

Architecture

The Alibaba Cloud Data Transmission Service (DTS) has an architectural flow that supports dual redundancy features. All of the functions of DTS are deployed on multiple servers that have dual redundancy. Let's take a look at each module to understand how it functions:

  1. HA Manager performs continuous health checks on each server. If any issue is detected, the workload shifts to the redundant service without any delay. Similarly, if there is an endpoint change detected, the HA manager reconfigures the data sources for proper functioning.
  2. Monitoring Server: The HA manager and the monitoring server are a part of the same module and work in sync with each other. The main difference is that the Monitoring server sends data back and forth to the DTS console and the scheduler.

Let's take a look at the architectural flow:
Architecture of Data Transmission Service

Use DTS for Real-time Data Synchronization between ApsaraDB RDS for MySQL and Alibaba Cloud Elasticsearch

This article describes how to use DTS to synchronize data from an ApsaraDB RDS for MySQL instance to an Alibaba Cloud Elasticsearch instance in real-time.

Data Transmission Service (DTS) synchronizes production data from an ApsaraDB RDS for MySQL instance to an Alibaba Cloud Elasticsearch instance in real-time after you create a real-time data synchronization task in the DTS console. This article focuses on the various supported real-time synchronization types and SQL operations. It further elaborates on the configuration procedure to support synchronization.

Supported Real-time Synchronization Types

Real-time data synchronization from an ApsaraDB RDS for MySQL instance to an Alibaba Cloud Elasticsearch instance in the same Alibaba Cloud account.

Supported SQL Operations

Supported SQL operations include:

1. Inset
2. Delete
3. Update

Note: Currently, DTS does not support Data Definition Language (DDL) synchronization and ignores DDL operations during synchronization.

In case a DDL operation of a table is encountered, the Data Manipulation Language (DML) operation of the table may fail. To rectify the failure, follow the steps below:

1) Delete the table from the synchronization object list. For more information, see "Remove an object from a data synchronization task.
2) Delete the index corresponding to this table in the Alibaba Cloud Elasticsearch instance.
3) Modify the synchronization task, re-add the table to the synchronization objects, and re-initialize the table. For more information, see "Add an object to a data synchronization task."

If a DDL operation is used to modify a table and add a column, the recommended DDL operation sequence is as follows:

1) Modify the mapping of the corresponding table and add a new column in the Alibaba Cloud Elasticsearch instance.
2) Modify the table structure and add a column in the source ApsaraDB RDS for MySQL instance.
3) Pause and restart the DTS synchronization task to allow DTS to reload the modified mapping in the Alibaba Cloud Elasticsearch instance.

Related Products

Data Transmission Service

Data Transmission Service (DTS) helps you migrate data between data storage types, such as relational database, NoSQL, and OLAP. The service supports homogeneous migrations as well as heterogeneous migration between different data storage types.
DTS also can be used for continuous data replication with high availability. In addition, DTS can help you subscribe to the change data function of ApsaraDB for RDS. With DTS, you can easily implement scenarios such as data migration, remote real time data backup, real time data integration and cache refresh.

Related Documentation

DTS Use cases

Data Transmission Service (DTS) supports data replication in several modes, including data migration, change tracking, and data synchronization. You can use a combination of data replication modes that best suit your use cases.

Supported sources and targets

This topic describes the supported data migration paths (source-target pairs) and replication features of all replication modes. For each migration path, the supported cloud-based and user-created databases and their corresponding replication features are listed.

dts data transmission service synchronization channel data synchronization types data migration mode
0 0 0
Share on

Read previous post:

Cloud-Native and Application Management with Ease – Part 7: The Evolution of Microservices with Cloud-Native Architecture

Read next post:

NoOps – the Reason for a Paradigm Shift in IT Operations

Alibaba Clouder

2,599 posts | 762 followers

You may also like

Comments

Alibaba Clouder

2,599 posts | 762 followers

Related Products

More Posts by Alibaba Clouder

See All