DataWorks provides Sensors Data Writer for you to write data to Sensors Data data sources. This topic describes the capabilities of synchronizing data to Sensors Data data sources.
Limits
Sensors Data data sources are supported only in the China (Shenzhen) region. They will be supported in other regions in the future.
DataWorks does not provide a dedicated Reader for reading data from Sensors Data data sources. You can use Hive Reader or HDFS Reader to read data based on the storage type of Sensors Data data sources.
Develop a data synchronization task
For information about the entry point for and the procedure of configuring a data synchronization task, see the following sections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.
Add a data source
Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.
Configure a batch synchronization task to synchronize data of a single table
For more information about the configuration procedure, see Configure a batch synchronization task by using the code editor.
For information about all parameters that are configured and the code that is run when you use the code editor to configure a batch synchronization task, see Appendix: Code and parameters.
Appendix: Code and parameters
Appendix: Configure a batch synchronization task by using the code editor
If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.
Code for Sensors Data Writer
{
"type": "job",
"steps": [
{
"stepType": "mysql",
"parameter": {},
"name": "Reader",
"category": "reader"
},
{
"stepType": "sahistory", // The plug-in name.
"parameter": {
"type": "item", // The type of data written to Sensors Data. Valid values: track, user, and item, which indicate events, users, and properties in Sensors Data.
"item": { // If you set the type parameter to item, define the item parameter.
"itemType": "course", // The item type. In this example, the itemType parameter is set to course.
"typeIsColumn": false, // Specifies whether the value of the itemType parameter needs to be specified as a column in the column parameter.
"itemIdColumn": "course_id" // The name of the field that stores the distinct ID of the property.
},
"column": [ // Specifies the mappings between fields in the source table and fields in the destination table.
{
"name": "course_id", // The name of a field in the destination table. In this example, the field name is course_id.
"index": 0 // Write data in the first field in the source table to the course_id field in the destination table.
},
{
"name": "course_name", // The name of a field in the destination table. In this example, the field name is course_name.
"index": 1 // Write data in the second field in the source table to the course_name field in the destination table.
},
{
"name": "course_schedule", // The name of a field in the destination table. In this example, the field name is course_schedule.
"index": 2 // Write data in the third field in the source table to the course_schedule field in the destination table.
"dataConverters":[ // The data type converters.
{
"type": "Long2Date" // The type of the data type converter.
}
]
}
],
"sdkDataAddress": "http://bigdata-project.datasink.sensorsdata.cn/sa?project=default&token=1111111111111111", // The Sensors Data URL to which the received data is stored.
},
"name": "Writer",
"category": "writer"
}
],
},
"setting": {
"errorLimit": {
"record": "0" // The maximum number of dirty data records allowed.
},
"speed": {
"throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true.
"concurrent": 2, // The maximum number of parallel threads.
"mbps":"12"// The maximum transmission rate. Unit: MB/s.
}
}
}
Parameters in code for Sensors Data Writer
Parameter | Description | Required | Default value |
type | The type of data written to a Sensors Data data source. Valid values: track, user, and item, which indicate events, users, and properties in Sensors Data. The value of the type parameter determines the parameters that you can use to define the data type. | Yes | No default value |
If you set the type parameter to track, you can configure the following parameters:
Sample code:
| |||
If you set the type parameter to user, you can configure the following parameters:
Sample code:
| |||
If you set the type parameter to item, you can configure the following parameters:
Sample code:
| |||
column | The mappings between fields in the source table and fields in the destination table. The mappings are determined by the name and index parameters.
| Yes | No default value |
sdkDataAddress | The URL to which the data is stored, which is the URL from which Sensors Data receives data. The URL is in the | Yes | No default value |
Other parameters
DataWorks allows you to specify data type converters when you configure Sensors Data Writer by using the code editor. You can specify the types of data type converters that you want to use to convert the data type of the data to be written to Sensors Data. The following table describes the types of converters that are supported and sample parameter settings.
Converter type | Feature description | Sample setting | Parameter description |
Date2Str | Converts data of the DATE type to data of the STRING type. |
| pattern (optional): The date format of the data after conversion. |
Date2Long | Converts data of the DATE type to data of the LONG type. |
| N/A. |
Number2Str | Converts data of the NUMBER type to data of the STRING type. |
| N/A. |
Str2Long | Converts data of the STRING type to data of the LONG type. |
| N/A. |
Str2Date | Converts data of the STRING type to data of the DATE type. |
|
|
BigInt2Date | Converts data of the BIGINT type to data of the DATE type. |
| N/A. |
Str2Int | Converts data of the STRING type to data of the INT type. |
| N/A. |
Str2Double | Converts data of the STRING type to data of the DOUBLE type. |
| N/A. |
Str2BigDecimal | Converts data of the STRING type to data of the BIGDECIMAL type. |
| N/A. |
IfNull2Default | Converts the NULL value to the constant value specified by the default parameter, and then converts the constant value to data of a specified data type by using the sub-converters embedded in the IfNull2Default converter. |
| dataConverters: the sub-converters that are used to convert the default constant value to data of a specified data type. You can specify multiple embedded sub-converters to perform data type conversions for multiple times. |
NotNull2Null | Converts a non-NULL value to the NULL value. |
| N/A. |
IfElse | Checks whether a field value meets a specified IF condition. If the field value meets the condition, the return value is indicated by the value parameter. Otherwise, the return value is indicated by the else parameter. |
|
|
IfNull2Column | Converts the NULL value to the value of the targetColumnName parameter. |
| targetColumnName: the name of the field in Sensors Data. Important Make sure that you configure the field specified by the targetColumnName parameter before you configure the fields to which the IfNull2Column converter applies. |