All Products
Search
Document Center

DataWorks:Sensors Data data source

Last Updated:Nov 13, 2023

DataWorks provides Sensors Data Writer for you to write data to Sensors Data data sources. This topic describes the capabilities of synchronizing data to Sensors Data data sources.

Limits

  • Sensors Data data sources are supported only in the China (Shenzhen) region. They will be supported in other regions in the future.

  • DataWorks does not provide a dedicated Reader for reading data from Sensors Data data sources. You can use Hive Reader or HDFS Reader to read data based on the storage type of Sensors Data data sources.

Develop a data synchronization task

For information about the entry point for and the procedure of configuring a data synchronization task, see the following sections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.

Add a data source

Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.

Configure a batch synchronization task to synchronize data of a single table

Appendix: Code and parameters

Appendix: Configure a batch synchronization task by using the code editor

If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.

Code for Sensors Data Writer

{
    "type": "job",
    "steps": [
        {
            "stepType": "mysql",
            "parameter": {},
            "name": "Reader",
            "category": "reader"
        },
        {
             "stepType": "sahistory", // The plug-in name. 
             "parameter": {
                "type": "item", // The type of data written to Sensors Data. Valid values: track, user, and item, which indicate events, users, and properties in Sensors Data. 
                     "item": {       // If you set the type parameter to item, define the item parameter. 
                    "itemType": "course", // The item type. In this example, the itemType parameter is set to course.
                    "typeIsColumn": false, // Specifies whether the value of the itemType parameter needs to be specified as a column in the column parameter.
                    "itemIdColumn": "course_id" // The name of the field that stores the distinct ID of the property.
                },
                "column": [     // Specifies the mappings between fields in the source table and fields in the destination table.
                    {
                        "name": "course_id",    // The name of a field in the destination table. In this example, the field name is course_id.
                        "index": 0              // Write data in the first field in the source table to the course_id field in the destination table.
                    },
                    {
                        "name": "course_name",  // The name of a field in the destination table. In this example, the field name is course_name.
                        "index": 1              // Write data in the second field in the source table to the course_name field in the destination table.
                    },
                    {
                        "name": "course_schedule",  // The name of a field in the destination table. In this example, the field name is course_schedule.
                        "index": 2                  // Write data in the third field in the source table to the course_schedule field in the destination table.
                        "dataConverters":[          // The data type converters. 
                               {
                                    "type": "Long2Date"   // The type of the data type converter. 
                                }
                          ]
                    }
                ],
                "sdkDataAddress": "http://bigdata-project.datasink.sensorsdata.cn/sa?project=default&token=1111111111111111", // The Sensors Data URL to which the received data is stored.
                  },
            "name": "Writer",
            "category": "writer"
        }
    ],
    },
    "setting": {
        "errorLimit": {
            "record": "0" // The maximum number of dirty data records allowed. 
        },
        "speed": {
            "throttle":true,// Specifies whether to enable throttling. The value false indicates that throttling is disabled, and the value true indicates that throttling is enabled. The mbps parameter takes effect only when the throttle parameter is set to true. 
                        "concurrent": 2, // The maximum number of parallel threads. 
                        "mbps":"12"// The maximum transmission rate. Unit: MB/s. 
        }
    }
}

Parameters in code for Sensors Data Writer

Parameter

Description

Required

Default value

type

The type of data written to a Sensors Data data source. Valid values: track, user, and item, which indicate events, users, and properties in Sensors Data. The value of the type parameter determines the parameters that you can use to define the data type.

Yes

No default value

If you set the type parameter to track, you can configure the following parameters:

  • distinctIdColumn (optional): The name of the field that stores the distinct ID of the event in Sensors Data. The field must also be specified in the column parameter as an attribute of the event, and the attribute value cannot be left empty.

  • eventName (optional): The name of the event in Sensors Data.

  • isLoginId (optional): Specifies whether the field that stores the distinct ID of the event also specifies the logon ID. A logon ID uniquely identifies a user who logs on to the Sensors Data platform. Valid values: true and false. Default value: true.

Sample code:

"track": {  
"distinctIdColumn": "track_id",  
"eventName": "testEventName", 
"isLoginId": true
}

If you set the type parameter to user, you can configure the following parameters:

  • distinctIdColumn (optional): The name of the field that stores the distinct ID of the user in Sensors Data. The field must also be specified in the column parameter as an attribute of the user, and the attribute value cannot be left empty.

  • isLoginId (optional): Specifies whether the field that stores the distinct ID of the event also specifies the logon ID. A logon ID uniquely identifies a user who logs on to the Sensors Data platform. Valid values: true and false. Default value: true.

Sample code:

"user": {
"distinctIdColumn": "user_id", 
"isLoginId": true
}

If you set the type parameter to item, you can configure the following parameters:

  • itemIdColumn (optional): The name of the field that stores the distinct ID of the property in Sensors Data. The field must also be specified in the column parameter as an attribute of the property, and the attribute value cannot be left empty.

  • itemType (optional): The name of the field that stores the item type of the property in Sensors Data. If the field is also specified in the column parameter, do not leave the field empty, and set the typeIsColumn parameter to true. Otherwise, a constant value is used as the item type.

  • typeIsColumn (optional): Specifies whether the value of the itemType parameter needs to be specified as a column in the column parameter. Valid values: true and false. Default value: true.

Sample code:

"item": {
"itemIdColumn": "item_id", 
"itemType": "testItem",
"typeIsColumn": false
}

column

The mappings between fields in the source table and fields in the destination table. The mappings are determined by the name and index parameters.

  • name (required): The name of the field in the destination table.

  • index (required): The index that indicates the sequence of the field in the source table. For example, the first field in the source table is indicated by the index 0.

Yes

No default value

sdkDataAddress

The URL to which the data is stored, which is the URL from which Sensors Data receives data. The URL is in the http://localhost:8106/sa?project=default format. To obtain the URL, log on to the Sensors Data platform and perform the following steps: In the top navigation bar, choose More > Basic Settings. On the Basic Settings tab, click Data Integration. In the Server tracking section of the page that appears, click Copy receive address of data.

Yes

No default value

Other parameters

DataWorks allows you to specify data type converters when you configure Sensors Data Writer by using the code editor. You can specify the types of data type converters that you want to use to convert the data type of the data to be written to Sensors Data. The following table describes the types of converters that are supported and sample parameter settings.

Converter type

Feature description

Sample setting

Parameter description

Date2Str

Converts data of the DATE type to data of the STRING type.

// Sample code
"dataConverters":[
    {
        "type": "Date2Str",
        "param": {
            "pattern":"yyyy-MM-dd"
        }
    }
]

pattern (optional): The date format of the data after conversion.

Date2Long

Converts data of the DATE type to data of the LONG type.

"dataConverters":[
    {
        "type": "Date2Long"
    }
]

N/A.

Number2Str

Converts data of the NUMBER type to data of the STRING type.

// Sample code
"dataConverters":[
    {
        "type": "Number2Str"
    }
]

N/A.

Str2Long

Converts data of the STRING type to data of the LONG type.

// Sample code
"dataConverters":[
    {
        "type": "Str2Long"
    }
]

N/A.

Str2Date

Converts data of the STRING type to data of the DATE type.

// Sample code
"dataConverters":[
    {
        "type": "Str2Date",
        "param": {
            "pattern":"yyyy-MM-dd",
            "formats":["yyyyMMdd","yyyyMMddHHmmss"]
        }
    }
]
  • pattern (optional): The date and time format of the data after conversion. Valid values: yyyy-MM-dd, yyyy-MM-dd HH:mm:ss, yyyy-MM-dd HH:mm:ss.SSS yyyy-MM, yyyyMM, yyyyMMddHHmmss, and yyyyMMddHHmmssSSS.

  • formats (optional): The supported date and time formats for data of the STRING type. This parameter allows you to specify more date and time formats if the values of the pattern parameter do not meet your requirements.

BigInt2Date

Converts data of the BIGINT type to data of the DATE type.

// Sample code
"dataConverters":[
    {
        "type": "BigInt2Date"
    }
]

N/A.

Str2Int

Converts data of the STRING type to data of the INT type.

// Sample code
"dataConverters":[
    {
        "type": "Str2Int"
    }
]

N/A.

Str2Double

Converts data of the STRING type to data of the DOUBLE type.

// Sample code
"dataConverters":[
    {
        "type": "Str2Double"
    }
]

N/A.

Str2BigDecimal

Converts data of the STRING type to data of the BIGDECIMAL type.

// Sample code
"dataConverters":[
    {
        "type": "Str2BigDecimal"
    }
]

N/A.

IfNull2Default

Converts the NULL value to the constant value specified by the default parameter, and then converts the constant value to data of a specified data type by using the sub-converters embedded in the IfNull2Default converter.

// Sample code
// Converts a NULL value or an empty string to the 2021-07-01 string, converts the string to time in the yyyy-MM-dd format, and then converts the time to the timestamp in milliseconds of the LONG type. 
"dataConverters":[
    {
        "type": "IfNull2Default",
        "param": {
            "default": "2021-07-01",
            "dataConverters": [
                {
                    "type": "Str2Date",
                    "param": {
                        "pattern":"yyyy-MM-dd"
                    }
                },
                {
                    "type": "Date2Long"
                }
            ]
        }
    }
]

dataConverters: the sub-converters that are used to convert the default constant value to data of a specified data type. You can specify multiple embedded sub-converters to perform data type conversions for multiple times.

NotNull2Null

Converts a non-NULL value to the NULL value.

// Sample code
"dataConverters":[
    {
        "type": "NotNull2Null"
    }
]

N/A.

IfElse

Checks whether a field value meets a specified IF condition. If the field value meets the condition, the return value is indicated by the value parameter. Otherwise, the return value is indicated by the else parameter.

// Sample code
"dataConverters":[
                  {
                    "type": "IfElse",
                    "param": {
                      "if": "return true;",
                      "value":"if(resolvedValues.get(\"check_card_type\") == 'NAN' && value == null ){return true;} if(resolvedValues.get(\"check_card_type\") == 'ID card'){  if(java.util.regex.Pattern.compile(regEx).matcher(value).find() && java.lang.Integer.parseInt(value.substring(6, 10)) >=1900 && java.lang.Integer.parseInt(value.substring(6, 10)) <= nowYear){return true;}} return null;",
                      "sharedPool":"var nowYear = java.time.LocalDate.now().getYear(); var regEx = \"(110|120|310|510|150|650|540|640|450|230|220|210|120|140|630|370|410|320|340|320|340|330|350|360|430|420|440|460|620|610|520|530)\\\\d{7}(0[1-9]|1[0-2])(0[1-9]|[1-2][0-9]|3[0-1])\\\\d{3}(\\\\d|X)\"; "
                    }
                  }
                ]
  • if: the IF condition. The information that you enter is parsed in JavaScript. Make sure that the result is of the BOOLEAN type. The field to which data is written in the Sensors Data data source, the value that is obtained by the plug-in, the param parameter of the data converter, and the parsed fields in the map data structure can be passed to the expression. You can use the targetColumnName, value, param, and resolvedValues parameters to obtain these values.

  • value: the return value if the IF condition is met. The information that you enter is parsed in JavaScript. The field to which data is written in the Sensors Data data source, the value that is obtained by the plug-in, the param parameter of the data converter, and the parsed fields in the map data structure can be passed to the expression. You can use the targetColumnName, value, param, and resolvedValues parameters to obtain these values.

  • else: the return value if the IF condition is not met. The information that you enter is parsed in JavaScript. The field to which data is written in the Sensors Data data source, the value that is obtained by the plug-in, the param parameter of the data converter, and the parsed fields in the map data structure can be passed to the expression. You can use the targetColumnName, value, param, and resolvedValues parameters to obtain these values.

  • sharedPool: the variables or constant values that can be used in the if, value, and else parameters. The information that you enter is parsed in JavaScript.

IfNull2Column

Converts the NULL value to the value of the targetColumnName parameter.

// Sample code
"dataConverters": [
    {
        "type": "IfNull2Column",
        "param": {
            "targetColumnName":"age1"
        }
    }
]

targetColumnName: the name of the field in Sensors Data.

Important

Make sure that you configure the field specified by the targetColumnName parameter before you configure the fields to which the IfNull2Column converter applies.