DataWorks:Configure a synchronization task in Data Integration

You can use one of the following methods to create a synchronization task:

• Method 1: Go to the Data Integration page. On this page, select the source type and destination type and click Create.

• Method 2: Go to the Data Integration page. If no synchronization tasks are displayed in the Nodes section of this page, click Create.

On the Create Data Synchronization Solution page, select the source and destination based on your business requirements. After you select the data source types, the system displays the supported synchronization task types in the Synchronization Method drop-down list. You can select a synchronization task type based on your business requirements.

Select the source, destination, and an exclusive resource group for Data Integration that you want to use. Then, test network connectivity between the resource group and data sources.

After you select a source database and a table, data is written to the destination schema and table that are named the same as the source database and table by default. If no such destination schema or table exists, the system automatically creates the schema or table in the destination. You can configure a mapping rule to specify the name of the destination schema or table to which you want to write data in the Set Mapping Rules for Table/Database Names section. You can specify a destination table name in a mapping rule to write data in multiple source tables to the same table. You can also specify prefixes in a mapping rule to write data to a database whose name starts with a different prefix from the source database or to tables whose names start with a different prefix from the source tables.

• Conversion Rule for Table Name: This type of mapping rule allows you to use a regular expression to map the names of the destination tables to which you want to write data to the names of source tables.

  • Example 1: Synchronize data from the source tables whose names start with the prefix doc_ to the destination tables whose names start with the prefix pre_.

  • Example 2: Synchronize data from multiple source tables to the same destination table.

    To synchronize data from table_01, table_02, and table_03 to my_table, you can configure a mapping rule of the Conversion Rule for Table Name type, and set Source to table.* and Destination to my_table.

• Rule for Destination Table Name: This type of mapping rule allows you to use built-in variables to specify the names of the destination tables to which you want to write data and add a prefix and a suffix to the names of the destination tables. The following built-in variables are supported:

  • ${db_table_name_src_transed}: the name of the destination table that is mapped based on a mapping rule of the Conversion Rule for Table Name type

  • ${db_name_src_transed}: the name of the destination schema that is mapped based on a mapping rule of the Rule for Conversion Between Source Database Name and Destination Schema Name type

  • ${ds_name_src}: the name of the source

  Example: Configure pre_${db_table_name_src_transed}_post to convert the table name my_table that is generated in the previous example to pre_my_table_post.

• Rule for Conversion Between Source Database Name and Destination Schema Name: This type of mapping rule allows you to use a regular expression to specify the names of the destination schemas to which you want to write data.

  Example: Synchronize data from the source schemas whose names start with the prefix doc_ to the destination schemas whose names start with the prefix pre_.

You can configure the properties related to the destination tables. For example, you can specify the write mode, whether to write data to partitioned destination tables, and the name of the partition key column. You can choose to write data to the existing destination tables or newly created destination tables. You can also add additional fields to the destination tables and assign values to the additional fields.

The parameters that you must configure vary based on the synchronization task type that you selected.

• Configure rules to process DDL or DML messages for a one-click real-time synchronization task

  DDL or DML operations may be performed on the source. To ensure that data synchronized to the destination meets your business requirements, you can configure rules to process DDL or DML messages from the source based on the destination type. For information about how to configure rules to process DDL messages or DML messages, see Configure rules to process DDL messages or Configure rules to process DML messages.

• Configure synchronization rules for a synchronization task that is used for batch synchronization of all data in a database

  When you configure a synchronization task that is used for batch synchronization of all data in a database, you must configure synchronization rules for the task. For example, you must specify a filter condition to implement incremental synchronization and configure scheduling settings for the task.

  • Implement incremental synchronization: You can use a WHERE clause to extract only incremental data from the source tables. You need to enter only the WHERE clause in the Condition for Incremental Synchronization field without the need to enter the WHERE keyword. When you specify the WHERE clause, you can use a built-in variable. For example, you can use ${bdp.system.bizdate} to represent the data timestamp of the task or use ${bdp.system.cyctime} to represent the scheduling time of the task.

    Note

    You can use scheduling parameters to specify the source tables from which you want to read data and the destination tables to which you want to write data. For more information about how to use scheduling parameters, see Description for using scheduling parameters in data synchronization.

  • Configure scheduling settings: A synchronization task that is used for batch synchronization of all data in a database must be scheduled on a regular basis. Before you run such a synchronization task to synchronize data, you must configure scheduling settings for the task such as the scheduling cycle, the effective date, and whether to pause scheduling. The scheduling settings for a synchronization task that is used for batch synchronization of all data in a database in Data Integration are the same as the scheduling settings for a synchronization node in DataStudio. For more information about the parameters that are required, see Configure time properties.

When you configure a synchronization task, you must configure names or naming rules for subtasks that will be generated by the synchronization task, and specify the resource groups that you want to use to run the subtasks. Data Integration provides default settings for the maximum numbers of connections and parallel threads that are allowed. You can click Advanced configuration to modify the default settings to meet your business requirements.

• If you create and run a one-click real-time synchronization task, the task will generate batch synchronization subtasks that are used to synchronize full data and real-time synchronization subtasks that are used to synchronize incremental data. You must configure properties related to the batch synchronization subtasks and real-time synchronization subtasks in the Run resource settings step. The properties include the names or naming rules of the real-time synchronization subtasks and batch synchronization subtasks, the exclusive resource groups for Data Integration used to run the real-time synchronization subtasks and batch synchronization subtasks, and the exclusive resource group for scheduling used to run the batch synchronization subtasks.

• If you create and run a synchronization task that is used for batch synchronization of all data in a database, you must specify a naming rule for the batch synchronization subtasks that will be generated by the task, and select the exclusive resource group for Data Integration and exclusive resource group for scheduling used to run the batch synchronization subtasks.