Before you configure a data synchronization task, you must configure the data source from which you want to read data and the data source to which you want to write data on the Data Source page. Data Integration supports dozens of data source types, such as MySQL, MaxCompute, Hologres, Object Storage Service (OSS), and Kafka. The synchronization operations that you can perform vary based on the data source type. You can select appropriate data sources when you configure the source and destination of a synchronization task based on your business requirements.
Supported data source types and synchronization operations
DataWorks Data Integration provides the batch synchronization feature, real-time synchronization feature, and full and incremental synchronization feature. You can determine the feature that you need to use to configure a synchronization task based on the data source types supported by the features.
The batch synchronization feature provides readers and writers for you to read data from and write data to data sources.
The real-time synchronization feature allows you to configure a real-time synchronization task by using different data sources to synchronize incremental data from a single table or all tables in a database in real time.
The full and incremental synchronization feature allows you to create synchronization tasks that are used to synchronize data between different data sources in various scenarios, such as batch synchronization of all data in a database, and one-time full synchronization and real-time incremental synchronization.
For information about the support of different types of destinations for synchronization of data changes generated by DDL and DML operations on sources in real-time synchronization scenarios, see Supported DML and DDL operations.
For information about how to configure a batch synchronization task, real-time synchronization task, or full and incremental synchronization task, see Overview of the batch synchronization feature, Overview of the real-time synchronization feature, or Overview of the full and incremental synchronization feature.
Data source type | Batch synchronization | Real-time synchronization | Full and incremental synchronization | |||||||
Read data from a single table | Write data of a single table to a destination | Read incremental data from a single table | Write incremental data of a single table to a destination | Read incremental data from all tables in a database | Write incremental data of all tables in a database to a destination | Read data from all tables in a database in offline mode | Write data of all tables in a database to a destination in offline mode | Read full data at a time and incremental data in real time from a table or all tables in a database | Write full data and incremental data of a table or all tables in a database to a destination | |
| - | - | - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | - | - | |
|
| - | - | - | - |
| - | - | - | |
|
| - |
| - |
|
| - | - |
| |
|
| - | - | - | - |
| - | - | - | |
|
| - | - |
| - | - | - |
| - | |
| - | - | - | - | - | - | - | - | - | |
| - | - | - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | - | - | |
|
|
|
| - |
| - | - | - |
| |
|
| - | - | - | - |
| - | - | - | |
|
| - | - | - | - | - | - | - | - | |
|
| - | - | - | - |
| - | - | - | |
|
| - | - | - | - |
| - |
| - | |
Elasticsearch data source |
|
| - |
| - | - | - |
| - |
|
|
| - | - | - | - | - | - | - | - | |
GBase8a data source |
|
| - | - | - | - | - | - | - | - |
HBase data source |
|
| - | - | - | - | - | - | - | - |
|
| - | - | - | - | - | - | - | - | |
Hive data source |
|
| - | - | - | - | - |
| - | - |
|
|
|
| - |
|
|
| - |
| |
| - | - | - | - | - | - | - | - | - | |
|
| - | - | - | - |
| - | - | - | |
|
|
|
| - |
| - | - | - |
| |
|
| - | - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | - | - | |
|
|
| - | - | - | - | - | - | - | |
|
| - |
| - |
| - |
| - |
| |
|
| - | - | - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | - | - | |
| - | - | - | - | - | - | - | - | - | |
|
| - | - |
| - | - | - |
| - | |
|
|
| - |
| - |
| - |
| - | |
- |
| - | - | - | - | - | - | - | - | |
|
| - | - | - | - |
| - |
| - | |
|
| - |
| - |
| - |
| - | - | |
|
| - |
| - |
| - | - | - | - | |
|
|
| - |
| - |
| - |
| - | |
|
| - | - | - | - |
| - |
| - | |
|
| - | - | - | - |
| - |
| - | |
- |
| - | - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | - | - | |
| - | - | - | - | - | - | - | - | - | |
|
| - | - | - | - | - | - | - | - | |
|
| - | - | - |
| - |
| - | - | |
|
| - | - | - | - |
| - | - | - | |
|
| - | - | - | - | - | - | - | - | |
| - | - | - | - | - | - | - | - | - | |
- |
| - | - | - | - | - | - | - | - | |
Vertica data source |
|
| - | - | - | - | - | - | - | - |
