Creates a synchronization task of a new version in Data Integration. The following types of synchronization tasks are supported: real-time synchronization of all data in a MySQL database to Hologres and batch synchronization of all data in a MySQL database to Hive.
Debugging
Request parameters
Parameter | Type | Required | Example | Description |
Action | String | Yes | CreateDIJob | The operation that you want to perform. Set the value to CreateDIJob. |
ProjectId | Long | No | 10000 | The DataWorks workspace ID. You can call the ListProjects operation to obtain the ID. |
JobName | String | No | mysql_to_holo_sync_8772 | The name of the synchronization task. |
Description | String | No | Data synchronization from MySQL to Hologres | The description of the synchronization task. |
MigrationType | String | No | FullAndRealtimeIncremental | The synchronization type. Valid values:
|
SourceDataSourceType | String | No | MySQL | The type of the source. Set the value to MySQL. |
DestinationDataSourceType | String | No | Hologres | The type of the destination. Valid values: Hologres and Hive. |
SourceDataSourceSettings | Array | No | The settings of the source. Only a single source is supported. | |
DataSourceName | String | No | mysql_datasource_1 | The name of the data source. |
DataSourceProperties | Map | No | TimeZone | The properties of the data source. |
String | No | TimeZone | The properties of the source. The properties of a MySQL data source include TimeZone and Encoding. | |
DestinationDataSourceSettings | Array | No | The settings of the destination. Only a single destination is supported. | |
DataSourceName | String | No | holo_datasource_1 | The name of the data source. |
DataSourceProperties | Map | No | TimeZone | The properties of the data source. |
String | No | TimeZone | The properties of the destination. No properties can be configured for a Hologres data source. | |
ResourceSettings | Object | No | The resource settings. | |
OfflineResourceSettings | Object | No | The resource used for batch synchronization. | |
ResourceGroupIdentifier | String | No | S_res_group_111_222 | The identifier of the resource group for Data Integration used for batch synchronization. |
RealtimeResourceSettings | Object | No | The resource used for real-time synchronization. | |
ResourceGroupIdentifier | String | No | S_res_group_111_222 | The identifier of the resource group for Data Integration used for real-time synchronization. |
TransformationRules | Array | No | The list of transformation rules for objects involved in the synchronization task. Each entry in the list defines a transformation rule. | |
RuleName | String | No | rename_rule_1 | The name of the rule. If the values of the RuleActionType parameter and the RuleTargetType parameter are the same for multiple transformation rules, you must make sure that the transformation rule names are unique. |
RuleActionType | String | No | Rename | The type of the action. Valid values:
|
RuleTargetType | String | No | Table | The type of the object on which you want to perform the action. Valid values:
|
RuleExpression | String | No | {"expression":"${srcDatasoureName}_${srcDatabaseName}"} | The expression of the rule. An expression must be a JSON string. Example of a renaming rule: {"expression":"${srcDatasourceName}_${srcDatabaseName}_0922","variables":[{"variableName":"srcDatabaseName","variableRules":[{"from":"fromdb","to":"todb"}]}]}
Example of a rule used to add a specific field to the destination and assign a value to the field: {"columns":[{"columnName":"my_add_column","columnValueType":"Constant","columnValue":"123"}]}
Example of a rule used to specify primary key fields for a destination table: {"columns":["ukcolumn1","ukcolumn2"]}
Example of a rule used to process DML messages: {"dmlPolicies":[{"dmlType":"Delete","dmlAction":"Filter","filterCondition":"id > 1"}]}
|
TableMappings | Array | No | The list of mappings between rules used to select synchronization objects in the source and transformation rules applied to the selected synchronization objects. Each entry in the list displays a mapping between a rule used to select synchronization objects and a transformation rule applied to the selected synchronization objects. | |
SourceObjectSelectionRules | Array | No | The rule used to select synchronization objects in the source. You can configure multiple rules. | |
ObjectType | String | No | Table | The type of the object. Valid values:
|
Expression | String | No | mysql_table_1 | The expression. |
TransformationRules | Array | No | The transformation rules applied to the selected synchronization objects. | |
RuleName | String | No | rename_rule_1 | The name of the rule. If the values of the RuleActionType parameter and the RuleTargetType parameter are the same for multiple transformation rules, you must make sure that the transformation rule names are unique. |
RuleActionType | String | No | Rename | The type of the action. Valid values:
|
RuleTargetType | String | No | Table | The type of the object on which you want to perform the action. Valid values:
|
JobSettings | Object | No | The settings for the dimension of the synchronization task. The settings include processing policies for DDL messages, policies for data type mappings between source fields and destination fields, and runtime parameters of the synchronization task. | |
DdlHandlingSettings | Array | No | The processing settings for DDL messages. | |
Type | String | No | AddColumn | The type of the DDL operation. Valid values:
|
Action | String | No | Critical | The processing policy for DDL messages. Valid values:
|
ColumnDataTypeSettings | Array | No | The settings for data type mappings between source fields and destination fields. The value of this parameter must be an array. | |
SourceDataType | String | No | bigint | The data type of a source field. |
DestinationDataType | String | No | text | The data type of a destination field. |
RuntimeSettings | Array | No | The runtime settings. The value of this parameter must be an array. | |
Name | String | No | runtime.offline.concurrent | The name of the configuration item. Valid values:
|
Value | String | No | 1 | The value of the configuration item. |
CycleScheduleSettings | Object | No | The settings for periodic scheduling. | |
CycleMigrationType | String | No | Full | The synchronization type that requires periodic scheduling. Valid values:
|
ScheduleParameters | String | No | bizdate=$bizdate | The scheduling parameters. |
ChannelSettings | String | No | {"structInfo":"MANAGED","storageType":"TEXTFILE","writeMode":"APPEND","partitionColumns":[{"columnName":"pt","columnType":"STRING","comment":""}],"fieldDelimiter":""} | The channel control settings for the synchronization task. The value of this parameter must be a JSON string. |
ImportRuleSettings | Object | No | The import settings for the synchronization task. | |
Source | String | No | Datastudio | The import source of the task. Set the value to Datastudio, which indicates synchronization tasks created in DataStudio. |
FileId | String | No | 10000 | The ID of the task to be imported. |
SystemDebug | String | No | false | Specifies whether to perform system debugging. Valid values: true and false. Default value: false. |
Response parameters
Parameter | Type | Example | Description |
RequestId | String | 4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB | The request ID. You can locate logs and troubleshoot issues based on the ID. |
DIJobId | Long | 11792 | The synchronization task ID. |
Examples
Sample requests
http(s)://[Endpoint]/?Action=CreateDIJob
&ProjectId=10000
&JobName=mysql_to_holo_sync_8772
&Description=Data synchronization from MySQL to Hologres
&MigrationType=FullAndRealtimeIncremental
&SourceDataSourceType=MySQL
&DestinationDataSourceType=Hologres
&SourceDataSourceSettings=[{"DataSourceName":"mysql_datasource_1","DataSourceProperties":{"key":"TimeZone"}}]
&DestinationDataSourceSettings=[{"DataSourceName":"holo_datasource_1","DataSourceProperties":{"key":"TimeZone"}}]
&ResourceSettings={"OfflineResourceSettings":{"ResourceGroupIdentifier":"S_res_group_111_222"},"RealtimeResourceSettings":{"ResourceGroupIdentifier":"S_res_group_111_222"}}
&TransformationRules=[{"RuleName":"rename_rule_1","RuleActionType":"Rename","RuleTargetType":"Table","RuleExpression":"{\"expression\":\"${srcDatasoureName}_${srcDatabaseName}\"}"}]
&TableMappings=[{"SourceObjectSelectionRules":[{"ObjectType":"Table","Expression":"mysql_table_1"}],"TransformationRules":[{"RuleName":"rename_rule_1","RuleActionType":"Rename","RuleTargetType":"Table"}]}]
&JobSettings={"DdlHandlingSettings":[{"Type":"AddColumn","Action":"Critical"}],"ColumnDataTypeSettings":[{"SourceDataType":"bigint","DestinationDataType":"text"}],"RuntimeSettings":[{"Name":"runtime.offline.concurrent","Value":"1"}]}
&SystemDebug=false
&<Common request parameters>
Sample success responses
XML
format
HTTP/1.1 200 OK
Content-Type:application/xml
<CreateDIJobResponse>
<RequestId>4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB</RequestId>
<DIJobId>11792</DIJobId>
</CreateDIJobResponse>
JSON
format
HTTP/1.1 200 OK
Content-Type:application/json
{
"RequestId" : "4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB",
"DIJobId" : 11792
}
Error codes
HTTP status code | Error code | Error message | Description |
429 | Throttling.Api | The request for this resource has exceeded your available limit. | The number of requests for the resource has exceeded the upper limit. |
429 | Throttling.System | The DataWorks system is busy. Try again later. | The DataWorks system is busy. Try again later. |
429 | Throttling.User | Your request is too frequent. Try again later. | Excessive requests have been submitted within a short period of time. Try again later. |
500 | InternalError.System | An internal system error occurred. Try again later. | An internal error has occurred. Try again later. |
500 | InternalError.UserId.Missing | An internal system error occurred. Try again later. | An internal error has occurred. Try again later. |
For a list of error codes, see Service error codes.