DataWorks:Configure full and incremental synchronization

Actions

Description

Refresh Mapping Results

The system lists the selected source tables. Target table attributes take effect only after you refresh and confirm.

• Select the tables to synchronize and click Batch Refresh Mapping.

• Target Table Name: The target table name is automatically generated based on Customize Mapping Rules for Destination Table Names rules. The default format is ${Source_Database_Name}_${Table_Name}. If the target table does not exist, the system creates it automatically.

Customize Mapping Rules for Destination Table Names (Optional)

The system uses a default table name generation rule: ${Source_Database_Name}_${Table_Name}. You can also click Edit in the Customize Mapping Rules for Destination Table Names column to add custom target table name rules.

• Rule Name: Enter a name that clearly describes the business logic.

• Target Table Name: You can generate the target table name by clicking the button and selecting Manual Input or Built-in Variable for concatenation. Supported variables include Source Data Source Name, Source Database Name, and Source Table Name.

• Edit Built-in Variable: Built-in variables support string transformation based on the original value.

You can:

1. Add prefix/suffix to name: Add a prefix or suffix to the source table name by defining a constant.

   Rule configuration	Application effect

2. Unified string replacement: Replace the string dev_ in source table names with prd_.

   Rule configuration	Application effect

3. Merge multiple tables into one table.

   Rule configuration	Application effect

Edit Mapping of Field Data Types (Optional)

The system provides default mapping between source and target field types. You can click Edit Mapping of Field Data Types in the upper-right corner of the table to customize the field type mapping relationship between source and target tables, then click Apply and Refresh Mapping.

When editing field type mapping, ensure the field type conversion rules are correct; otherwise, type conversion failures may occur, leading to dirty data and affecting task execution.

Edit Destination Table Structure (Optional)

The system automatically creates non-existent target tables or reuses existing tables with the same name based on custom table name mapping rules.

DataWorks automatically generates the target table schema based on the source table schema. Manual intervention is not required for standard scenarios. You can also modify the table schema using the following methods:

• Add field to single table: Click the button in the Destination Table Name column to add a field.

• Batch add fields: Select all tables to be synchronized, and choose Batch Modify > Destination Table Structure - Batch Modify and Add Fields at the bottom of the table.

• Renaming columns is not supported.

For existing tables, you can only add fields. For new tables, you can add fields, partition fields, and set table types or properties. See the editing area in the interface for details.

Destination Table Column Assignment

Native fields are automatically mapped based on matching field names in the source and target tables. The Added Fields from the previous step require manual assignment. The procedure is as follows:

• Single table assignment: Click the Configure button in the Destination Table Column Assignment column to assign values to target table fields.

• Batch assignment: Select Batch Modify > Destination Table Column Assignment at the bottom of the list to batch assign values to identical fields in target tables.

You can assign constants or variables. Switch the type in Assignment Method. Supported methods include:

Table Field

• Manual assignment: Directly enter a constant value, such as: abc.

• Select variable: Select a supported system variable from the drop-down list. View specific variable meanings in the tooltip.

Source Split PK

You can select a field from the source table in the source split key drop-down list or select No Split. During synchronization, the task is split into multiple subtasks based on this field to enable concurrent and batch data reading.

We recommend using the primary key or a field with even data distribution as the source split key. String, float, and date types are not supported.

Source split keys are currently supported only for MySQL sources.

Execute Full Synchronization

If full synchronization was configured in Step 3, you can individually deselect full data synchronization for specific tables. This applies to scenarios where full data has already been synchronized to the target via other means.

Full Synchronization Condition

Filters source data during the full synchronization phase. Enter the WHERE clause excluding the WHERE keyword.

DML Rule

DML message processing is used to perform granular filtering and control on change data (Insert, Update, Delete) captured from the source before writing it to the target. This rule only takes effect during the incremental phase.

Full Data Merge Cycle

Currently only supports daily merge.

Merge Primary Key

You can define the primary key by selecting one or more columns from the table.

• Single column primary key: Select a single column with unique values (such as id) as the primary key.

• Composite primary key: If there is no unique single column, you can select a combination of multiple columns. The combined values of these columns must uniquely identify each row of data to ensure uniqueness when writing data and prevent overwriting.