All Products
Search
Document Center

:CreateDIJob

最終更新日:May 17, 2024

Creates a synchronization task of a new version in Data Integration. The following types of synchronization tasks are supported: real-time synchronization of all data in a MySQL database to Hologres and batch synchronization of all data in a MySQL database to Hive.

Debugging

OpenAPI Explorer automatically calculates the signature value. For your convenience, we recommend that you call this operation in OpenAPI Explorer. OpenAPI Explorer dynamically generates the sample code of the operation for different SDKs.

Request parameters

Parameter

Type

Required

Example

Description

Action

String

Yes

CreateDIJob

The operation that you want to perform. Set the value to CreateDIJob.

ProjectId

Long

No

10000

The DataWorks workspace ID. You can call the ListProjects operation to obtain the ID.

JobName

String

No

mysql_to_holo_sync_8772

The name of the synchronization task.

Description

String

No

Data synchronization from MySQL to Hologres

The description of the synchronization task.

MigrationType

String

No

FullAndRealtimeIncremental

The synchronization type. Valid values:

  • FullAndRealtimeIncremental (one-time full synchronization and real-time incremental synchronization)

  • RealtimeIncremental (real-time incremental synchronization)

  • Full (full synchronization)

  • OfflineIncremental (batch incremental synchronization)

  • FullAndOfflineIncremental (one-time full synchronization and batch incremental synchronization)

SourceDataSourceType

String

No

MySQL

The type of the source. Set the value to MySQL.

DestinationDataSourceType

String

No

Hologres

The type of the destination. Valid values: Hologres and Hive.

SourceDataSourceSettings

Array

No

The settings of the source. Only a single source is supported.

DataSourceName

String

No

mysql_datasource_1

The name of the data source.

DataSourceProperties

Map

No

TimeZone

The properties of the data source.

String

No

TimeZone

The properties of the source. The properties of a MySQL data source include TimeZone and Encoding.

DestinationDataSourceSettings

Array

No

The settings of the destination. Only a single destination is supported.

DataSourceName

String

No

holo_datasource_1

The name of the data source.

DataSourceProperties

Map

No

TimeZone

The properties of the data source.

String

No

TimeZone

The properties of the destination. No properties can be configured for a Hologres data source.

ResourceSettings

Object

No

The resource settings.

OfflineResourceSettings

Object

No

The resource used for batch synchronization.

ResourceGroupIdentifier

String

No

S_res_group_111_222

The identifier of the resource group for Data Integration used for batch synchronization.

RealtimeResourceSettings

Object

No

The resource used for real-time synchronization.

ResourceGroupIdentifier

String

No

S_res_group_111_222

The identifier of the resource group for Data Integration used for real-time synchronization.

TransformationRules

Array

No

The list of transformation rules for objects involved in the synchronization task. Each entry in the list defines a transformation rule.

RuleName

String

No

rename_rule_1

The name of the rule. If the values of the RuleActionType parameter and the RuleTargetType parameter are the same for multiple transformation rules, you must make sure that the transformation rule names are unique.

RuleActionType

String

No

Rename

The type of the action. Valid values:

  • DefinePrimaryKey

  • Rename

  • AddColumn

  • HandleDml

  • DefineIncrementalCondition

  • DefineCycleScheduleSettings

  • DefineRuntimeSettings

  • DefinePartitionKey

RuleTargetType

String

No

Table

The type of the object on which you want to perform the action. Valid values:

  • Table

  • Schema

RuleExpression

String

No

{"expression":"${srcDatasoureName}_${srcDatabaseName}"}

The expression of the rule. An expression must be a JSON string.

Example of a renaming rule:

{"expression":"${srcDatasourceName}_${srcDatabaseName}_0922","variables":[{"variableName":"srcDatabaseName","variableRules":[{"from":"fromdb","to":"todb"}]}]}

  • expression: the expression of the renaming rule. You can use the following variables in an expression: ${srcDatasourceName}, ${srcDatabaseName}, and ${srcTableName}. ${srcDatasourceName} indicates the name of the source. ${srcDatabaseName} indicates the name of a source database. ${srcTableName} indicates the name of a source table.

  • variables: the generation rule for a variable used in the expression of the renaming rule. The default value of the specified variable is the original value of the object indicated by the variable. You can define a group of string replacement rules to change the original values based on your business requirements. variableName: the name of the variable. Do not enclose the variable name in ${}. variableRules: the string replacement rules for variables. The system runs the string replacement rules in sequence for string replacement. from specifies the original string. to specifies the new string.

Example of a rule used to add a specific field to the destination and assign a value to the field:

{"columns":[{"columnName":"my_add_column","columnValueType":"Constant","columnValue":"123"}]}

  • If you do not configure such a rule, no fields are added to the destination and no values are assigned by default.

  • columnName: the name of the field that you want to add.

  • columnValueType: the type of the value of the field. Valid values: Constant and Variable.

  • columnValue: the value of the field that you want to add. If you set the valueType parameter to Constant, set the columnValue parameter to a custom constant of the STRING type. If you set the valueType parameter to Variable, set the columnValue to a built-in variable. The following built-in variables are supported: EXECUTE_TIME (LONG data type), DB_NAME_SRC (STRING data type), DATASOURCE_NAME_SRC (STRING data type), TABLE_NAME_SRC (STRING data type), DB_NAME_DEST (STRING data type), DATASOURCE_NAME_DEST (STRING data type), TABLE_NAME_DEST (STRING data type), and DB_NAME_SRC_TRANSED (STRING data type). EXECUTE_TIME specifies the execution time. DB_NAME_SRC specifies the name of a source database. DATASOURCE_NAME_SRC specifies the name of the source. TABLE_NAME_SRC specifies the name of a source table. DB_NAME_DEST specifies the name of a destination database. DATASOURCE_NAME_DEST specifies the name of the destination. TABLE_NAME_DEST specifies the name of a destination table. DB_NAME_SRC_TRANSED specifies the database name obtained after a transformation.

Example of a rule used to specify primary key fields for a destination table:

{"columns":["ukcolumn1","ukcolumn2"]}

  • If you do not configure such a rule, the primary key fields in the mapped source table are used for the destination table by default.

  • If the destination table is an existing table, Data Integration does not modify the schema of the destination table. If the specified primary key fields do not exist in the destination table, an error is reported when the synchronization task starts to run.

  • If the destination table is automatically created by the system, Data Integration automatically creates the schema of the destination table. The schema contains the primary key fields that you specify. If the specified primary key fields do not exist in the destination table, an error is reported when the synchronization task starts to run.

Example of a rule used to process DML messages:

{"dmlPolicies":[{"dmlType":"Delete","dmlAction":"Filter","filterCondition":"id > 1"}]}

  • If you do not configure such a rule, the default processing policy for messages generated for insert, update, and delete operations is Normal.

  • dmlType: the DML operation. Valid values: Insert, Update, and Delete.

  • dmlAction: the processing policy for DML messages. Valid values: Normal, Ignore, Filter, and LogicalDelete. Filter indicates conditional processing. You can set the dmlAction parameter to Filter only when the dmlType parameter is set to Update or Delete.

  • filterCondition: the condition used to filter DML messages. This parameter is required only when the dmlAction parameter is set to Filter.

TableMappings

Array

No

The list of mappings between rules used to select synchronization objects in the source and transformation rules applied to the selected synchronization objects. Each entry in the list displays a mapping between a rule used to select synchronization objects and a transformation rule applied to the selected synchronization objects.

SourceObjectSelectionRules

Array

No

The rule used to select synchronization objects in the source. You can configure multiple rules.

ObjectType

String

No

Table

The type of the object. Valid values:

  • Table

  • Database

Expression

String

No

mysql_table_1

The expression.

TransformationRules

Array

No

The transformation rules applied to the selected synchronization objects.

RuleName

String

No

rename_rule_1

The name of the rule. If the values of the RuleActionType parameter and the RuleTargetType parameter are the same for multiple transformation rules, you must make sure that the transformation rule names are unique.

RuleActionType

String

No

Rename

The type of the action. Valid values:

  • DefinePrimaryKey

  • Rename

  • AddColumn

  • HandleDml

  • DefineIncrementalCondition

  • DefineCycleScheduleSettings

  • DefineRuntimeSettings

  • DefinePartitionKey

RuleTargetType

String

No

Table

The type of the object on which you want to perform the action. Valid values:

  • Table

  • Schema

JobSettings

Object

No

The settings for the dimension of the synchronization task. The settings include processing policies for DDL messages, policies for data type mappings between source fields and destination fields, and runtime parameters of the synchronization task.

DdlHandlingSettings

Array

No

The processing settings for DDL messages.

Type

String

No

AddColumn

The type of the DDL operation. Valid values:

  • RenameColumn

  • ModifyColumn

  • CreateTable

  • TruncateTable

  • DropTable

  • DropColumn

  • AddColumn

Action

String

No

Critical

The processing policy for DDL messages. Valid values:

  • Ignore: ignores a DDL message.

  • Critical: reports an error for a DDL message.

  • Normal: normally processes a DDL message.

ColumnDataTypeSettings

Array

No

The settings for data type mappings between source fields and destination fields. The value of this parameter must be an array.

SourceDataType

String

No

bigint

The data type of a source field.

DestinationDataType

String

No

text

The data type of a destination field.

RuntimeSettings

Array

No

The runtime settings. The value of this parameter must be an array.

Name

String

No

runtime.offline.concurrent

The name of the configuration item. Valid values:

  • runtime.offline.speed.limit.mb: indicates the maximum transmission rate that is allowed for a batch synchronization task. This configuration item takes effect only when runtime.offline.speed.limit.enable is set to true.

  • runtime.offline.speed.limit.enable: indicates whether throttling is enabled for a batch synchronization task.

  • dst.offline.connection.max: indicates the maximum number of connections that are allowed for writing data to the destination of a batch synchronization task.

  • runtime.offline.concurrent: indicates the maximum number of parallel threads that are allowed for a batch synchronization task.

  • dst.realtime.connection.max: indicates the maximum number of connections that are allowed for writing data to the destination of a real-time synchronization task.

  • runtime.enable.auto.create.schema: indicates whether schemas are automatically created in the destination of a synchronization task.

  • src.offline.datasource.max.connection: indicates the maximum number of connections that are allowed for reading data from the source of a batch synchronization task.

  • runtime.realtime.concurrent: indicates the maximum number of parallel threads that are allowed for a real-time synchronization task.

Value

String

No

1

The value of the configuration item.

CycleScheduleSettings

Object

No

The settings for periodic scheduling.

CycleMigrationType

String

No

Full

The synchronization type that requires periodic scheduling. Valid values:

  • Full: full synchronization

  • OfflineIncremental: batch incremental synchronization

ScheduleParameters

String

No

bizdate=$bizdate

The scheduling parameters.

ChannelSettings

String

No

{"structInfo":"MANAGED","storageType":"TEXTFILE","writeMode":"APPEND","partitionColumns":[{"columnName":"pt","columnType":"STRING","comment":""}],"fieldDelimiter":""}

The channel control settings for the synchronization task. The value of this parameter must be a JSON string.

ImportRuleSettings

Object

No

The import settings for the synchronization task.

Source

String

No

Datastudio

The import source of the task.

Set the value to Datastudio, which indicates synchronization tasks created in DataStudio.

FileId

String

No

10000

The ID of the task to be imported.

SystemDebug

String

No

false

Specifies whether to perform system debugging. Valid values: true and false. Default value: false.

Response parameters

Parameter

Type

Example

Description

RequestId

String

4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB

The request ID. You can locate logs and troubleshoot issues based on the ID.

DIJobId

Long

11792

The synchronization task ID.

Examples

Sample requests

http(s)://[Endpoint]/?Action=CreateDIJob
&ProjectId=10000
&JobName=mysql_to_holo_sync_8772
&Description=Data synchronization from MySQL to Hologres
&MigrationType=FullAndRealtimeIncremental
&SourceDataSourceType=MySQL
&DestinationDataSourceType=Hologres
&SourceDataSourceSettings=[{"DataSourceName":"mysql_datasource_1","DataSourceProperties":{"key":"TimeZone"}}]
&DestinationDataSourceSettings=[{"DataSourceName":"holo_datasource_1","DataSourceProperties":{"key":"TimeZone"}}]
&ResourceSettings={"OfflineResourceSettings":{"ResourceGroupIdentifier":"S_res_group_111_222"},"RealtimeResourceSettings":{"ResourceGroupIdentifier":"S_res_group_111_222"}}
&TransformationRules=[{"RuleName":"rename_rule_1","RuleActionType":"Rename","RuleTargetType":"Table","RuleExpression":"{\"expression\":\"${srcDatasoureName}_${srcDatabaseName}\"}"}]
&TableMappings=[{"SourceObjectSelectionRules":[{"ObjectType":"Table","Expression":"mysql_table_1"}],"TransformationRules":[{"RuleName":"rename_rule_1","RuleActionType":"Rename","RuleTargetType":"Table"}]}]
&JobSettings={"DdlHandlingSettings":[{"Type":"AddColumn","Action":"Critical"}],"ColumnDataTypeSettings":[{"SourceDataType":"bigint","DestinationDataType":"text"}],"RuntimeSettings":[{"Name":"runtime.offline.concurrent","Value":"1"}]}
&SystemDebug=false
&<Common request parameters>

Sample success responses

XML format

HTTP/1.1 200 OK
Content-Type:application/xml

<CreateDIJobResponse>
    <RequestId>4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB</RequestId>
    <DIJobId>11792</DIJobId>
</CreateDIJobResponse>

JSON format

HTTP/1.1 200 OK
Content-Type:application/json

{
  "RequestId" : "4F6AB6B3-41FB-5EBB-AFB2-0C98D49DA2BB",
  "DIJobId" : 11792
}

Error codes

HTTP status code

Error code

Error message

Description

429

Throttling.Api

The request for this resource has exceeded your available limit.

The number of requests for the resource has exceeded the upper limit.

429

Throttling.System

The DataWorks system is busy. Try again later.

The DataWorks system is busy. Try again later.

429

Throttling.User

Your request is too frequent. Try again later.

Excessive requests have been submitted within a short period of time. Try again later.

500

InternalError.System

An internal system error occurred. Try again later.

An internal error has occurred. Try again later.

500

InternalError.UserId.Missing

An internal system error occurred. Try again later.

An internal error has occurred. Try again later.

For a list of error codes, see Service error codes.