DataWorks:Configure and manage a real-time synchronization node

Limits

• You can use only exclusive resource groups for Data Integration to run real-time synchronization nodes.

• You can use a real-time synchronization node to synchronize data to a Hologres data source only from a PolarDB, Oracle, or MySQL data source.

• A real-time data synchronization node cannot be used to synchronize data from a table that has no primary key.

Create a real-time synchronization node

1. Create a real-time synchronization node to synchronize all data in a database.
2. Configure an exclusive resource group for Data Integration.
3. Configure the source and mapping rules.
   1. In the Data Source section of the Configure Source and Synchronization Rules step, configure the Type and Data source parameters.
   2. Select the tables from which you want to read data.
      In the Source Table section, all tables in the selected data source are displayed in the Source Table list. You can select all or some tables from the Source Table list and click the icon to move the tables to the Selected Source Table list.

      Important If a selected table does not have a primary key, the table cannot be synchronized in real time.
   3. In the Conversion Rule for Table Name section, click Add Rule, select a rule type, and then configure a mapping rule based on rule type that you selected.
      By default, data in the source tables is written to the destination Hologres schemas or tables that are named the same as the source tables. You can specify a destination schema name and a destination table name in a mapping rule to write data in multiple source tables to the same Hologres table. You can also specify prefixes in a mapping rule to write data from source tables whose names start with a specified prefix to Hologres tables whose names start with another specified prefix. Data Integration allows you to use a regular expression to configure a mapping rule to specify the names of the destination Hologres schemas or tables to which you want to write data. You can also concatenate built-in variables to specify the names of the destination Hologres tables. For more information about the configuration logic, see Configure the source and mapping rules.
4. Configure the destination tables.
   1. Configure the Policy for Writing to Hologres parameter.
      You can set the Policy for Writing to Hologres parameter only to Replay. This value indicates that the operations performed on the source are also performed on the destination by Hologres Writer. The operations include INSERT, UPDATE, and DELETE.
   2. Refresh mappings between source tables and destination Hologres tables.
      Click Refresh source table and Hologres Table mapping to map the source tables and destination Hologres tables based on the mapping rules that you configured in the Conversion Rule for Table Name section. If no mapping rule is configured in the Conversion Rule for Table Name section, data in the source tables is written to the Hologres tables that are named the same as the source tables. If no such destination Hologres table exists in the destination, the system automatically creates the tables in the destination. You can modify the table generation method and add additional fields to the destination Hologres tables.

      Operation	Description
      Synchronize a source table that does not have a primary key	The current solution cannot be used to synchronize data from a source table that does not have a primary key. If you want to synchronize data from a source table that does not have a primary key, you must click the icon in the Synchronized Primary Key column of the table to specify a primary key for the source table. You can use a field or a combination of multiple fields in the source table as the primary key. The system removes duplicate data based on the primary key during data synchronization.
      Select a table generation method	You can select Create Table or Use Existing Table from the drop-down list in the Table creation method column. If you select Use Existing Table from the drop-down list in the Table creation method column, you can select a destination table from the drop-down list in the Hologres Table column. If you select Create Table from the drop-down list in the Table creation method column, the name of the table that is automatically created appears in the Hologres Table column. You can click the table name to view and modify the table creation statement.
      Add additional fields to a destination Hologres table and assign values to the fields	You can click Edit additional fields in the Actions column of a destination Hologres table to add additional fields to the table and assign values to the fields. You can manually assign constants and variables to the additional fields. Note You can add additional fields to a destination Hologres table only if you select Create Table from the drop-down list in the Table creation method column of the table. The following additional variable fields are supported by Data Integration: EXECUTE_TIME: the execution time UPDATE_TIME: the update time DB_NAME_SRC: the name of the original database DB_NAME_SRC_TRANSED: the converted name of the database DATASOURCE_NAME_SRC: the name of the source data source DATASOURCE_NAME_DEST: the name of the destination data source DB_NAME_DEST: the name of the destination database TABLE_NAME_DEST: the name of the destination table TABLE_NAME_SRC: the name of the source table

   3. Click Next.
      If you select Create Table from the drop-down list in the Table creation method column of a destination table, you must click Start table building in the Create Table dialog box to create destination Hologres tables.
5. Configure rules for processing DML messages.
   You can configure rules for processing DML messages generated for insert, update, and delete operations that are performed on the source.

   • Normal: A DML message from the source is sent to the destination. Then, the system performs the same operation on the destination as the operation that is performed on the source.
   • Ignore: A DML message from the source is ignored and is not sent to the destination. The system does not perform any operation on the destination.
   • Conditionally Normal Processing: If you configure a processing rule of this type, you can specify a filter condition to filter source data. The related operation will be performed only on data that meets the filter condition. Data that does not meet the filter condition will be ignored.
6. Configure rules for processing DDL messages.

   DDL operations are performed on a source. Data Integration provides default rules to process DDL messages. You can also configure processing rules for different DDL messages based on your business requirements. For more information, see Rules for processing DDL messages.

7. Configure the resources required to run the data synchronization node.
   1. In the Configure Resources step, configure the parameters.

      Parameter	Description
      Maximum number of connections supported by source read	The maximum number of Java Database Connectivity (JDBC) connections that are allowed for the source. Configure this parameter based on the resources of the source database. Default value: 15.
      Maximum number of parallel threads allowed to read by destination	The maximum number of parallel threads that the synchronization node uses to read data from the source table or write data to the destination. Maximum value: 32. Specify an appropriate number based on the specifications of the exclusive resource group for Data Integration and the data write capabilities of the destination.

   2. Click Complete Configuration.

Commit and deploy the real-time synchronization node

1. Click the icon in the top toolbar to save the node.
2. Click the icon in the top toolbar to commit the node.
3. In the Commit Node dialog box, configure the Change description parameter.
4. Click Confirm.
   If you use a workspace in standard mode, you must deploy the node in the production environment after you commit the node. On the left side of the top navigation bar, click Deploy. For more information, see Deploy nodes.