DataWorks:OSS data source

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.

Yes

No default value

Object

The name of the OSS object from which you want to read data. You can specify multiple object names. For example, a bucket has a directory named yunshi, and this directory contains an object named ll.txt. In this case, you can set this parameter to yunshi/ll.txt. This parameter can work with scheduling parameters to flexibly generate the name and path of an OSS object.

• If you specify a single OSS object name, OSS Reader uses only a single thread to read data. The feature of using parallel threads to read data from a single uncompressed object will be available in the future.

• If you specify multiple OSS object names, OSS Reader uses parallel threads to read data. You can configure the number of parallel threads based on your business requirements.

• If you specify a name that contains a wildcard, OSS Reader reads data from all objects that match the name. For example, if you set this parameter to abc*[0-9], OSS Reader reads data from objects such as abc0, abc1, abc2, and abc3. If you set this parameter to abc?.txt, OSS Reader reads data from objects whose names start with abc, end with .txt, and contain an arbitrary character between abc and .txt.

  We recommend that you do not use wildcards because an out of memory (OOM) error may occur. For more information, see What is OSS?

Yes

No default value

parquetSchema

The schema of Parquet files that you want to read. If you set the fileFormat parameter to parquet, you must configure the parquetSchema parameter. Make sure that the entire script complies with the JSON syntax.

message MessageTypeName {
required, dataType, columnName;
......................;
}

"parquetSchema": "message m { optional int32 minute_id; optional int32 dsp_id; optional int32 adx_pid; optional int64 req; optional int64 res; optional int64 suc; optional int64 imp; optional double revenue; }"

No

No default value

column

The names of the columns from which you want to read data. The type parameter specifies the source data type. The index parameter specifies the ID of the column in the source object, starting from 0. The value parameter specifies the column value if the column is a constant column. OSS Reader does not read a constant column from the source. Instead, OSS Reader generates a constant column based on the column value that you specify.

"column": ["*"]

"column":
    {
       "type": "long",
       "index": 0    // The first INT-type column in the object from which you want to read data. 
    },
    {
       "type": "string",
       "value": "alibaba"  // The value of the current column. In this code, the value is the constant alibaba. 
    }

Yes

"column": ["*"]

fileFormat

The format of the OSS object from which you want to read data. Examples: csv and text. OSS objects in the csv or text format support custom delimiters.

Yes

csv

fieldDelimiter

The column delimiter that is used in the OSS object from which you want to read data.

Yes

,

lineDelimiter

The row delimiter that is used in the OSS object from which you want to read data.

No

No default value

compress

The format in which objects are compressed. By default, this parameter is left empty, which indicates that objects are not compressed. OSS Reader supports the following compression formats: GZIP, BZIP2, and ZIP.

No

No default value

encoding

The encoding format of the object from which you want to read data.

No

utf-8

nullFormat

The string that represents a null pointer. No standard strings can represent a null pointer in TXT files. You can use this parameter to define a string that represents a null pointer. Examples:

• If you specify nullFormat:"null", the reader considers the printable string null as a null value.

• If you specify nullFormat:"\u0001", the reader considers the non-printable string \u0001 as a null value.

• If you do not configure the nullFormat parameter, OSS Reader does not convert source data.

No

No default value

skipHeader

Specifies whether to skip the headers in a CSV-like object if the object has headers. By default, the headers are not skipped. The skipHeader parameter is unavailable for compressed objects.

No

false

csvReaderConfig

The configurations required to read CSV objects. The parameter value must match the MAP type. You can use a CSV object reader to read data from CSV objects. The CSV object reader supports multiple configurations.

No

No default value

Parameter

Description

Required

Default value

datasource

The name of the data source. It must be the same as the name of the added data source. You can add data sources by using the code editor.

Yes

No default value

object

The prefix for the names of the files that you want to write to OSS. OSS simulates the directory effect by adding delimiters to file names. Examples:

• If you set the object parameter to datax, the names of the files start with datax and end with random strings.

• If you set the object parameter to cdo/datax, the names of the files start with /cdo/datax and end with random strings. OSS uses forward slashes (/) in file names to simulate the directory effect.

If you do not want to add a random universally unique identifier (UUID) as the suffix, we recommend that you set the writeSingleObject parameter to true. For more information, see the description of the writeSingleObject parameter.

Yes

No default value

writeMode

The write mode. Valid values:

• truncate: OSS Writer deletes all existing objects whose names start with the specified prefix before it writes files to OSS. For example, if you set the object parameter to abc, OSS Writer deletes all the objects whose names start with abc before it writes files to OSS.

• append: OSS Writer writes all files to OSS and suffixes the file names with random UUIDs to ensure that the names of the files are different from the names of existing objects. For example, if you set the object parameter to DI, the actual names of the files written to OSS are in the DI_****_****_**** format.

• nonConflict: If OSS contains objects whose names start with the specified prefix, OSS Writer returns an error. For example, if you set the object parameter to abc and OSS contains an object named abc123, OSS Writer returns an error.

Yes

No default value

writeSingleObject

Specifies whether to write a single file to OSS at a time. Valid values:

• true: writes a single file to OSS at a time. If no data is read, no empty file is generated.

• false: writes multiple files to OSS at a time. If no data is read and a file header is configured, an empty file that contains only the file header is generated. Otherwise, an empty file is generated.

No

false

fileFormat

The format in which OSS Writer writes files to OSS. Valid values:

• csv: If a file is written as a CSV file, the file must follow CSV specifications. If the data in the file contains column delimiters, the column delimiters are escaped by double quotation marks (").

• text: If a file is written as a text file, the data in the file is separated by column delimiters. In this case, OSS Writer does not escape the column delimiters.

• parquet: OSS Writer can write Parquet files to OSS. If you want to write Parquet files to OSS, you must configure the parquetschema parameter to define the related data type.

  Important

  • When you write data of the Parquet type, you must switch to the code editor and configure the parquetSchema parameter. For more information, see Appendix: Code and parameters.

  • If you do not configure the parquetSchema parameter, DataWorks converts the data types of source data. For more information about the conversion policy, see Appendix 2: Convert the data types of data in Parquet files.

• orc: If you want to write ORC files to OSS, you must use the code editor.

No

text

compress

The compression type of the files that you want to write to OSS. This parameter is available only in the code editor.

No

No default value

fieldDelimiter

The column delimiter that is used in the files that you want to write to OSS.

No

,

encoding

The encoding format of the files that you want to write to OSS.

No

utf-8

parquetSchema

The schema of the Parquet files that you want to write to OSS. If you set the fileFormat parameter to parquet, you must configure this parameter. Format:

message MessageTypeName {
required, dataType, columnName;
......................;
}

Fields:

• MessageTypeName: the name of the MessageType object.

• required: indicates that the column cannot be left empty. You can also specify optional based on your business requirements. We recommend that you specify optional for all columns.

• dataType: Parquet files support various data types, such as BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BINARY, and FIXED_LEN_BYTE_ARRAY. Set this parameter to BINARY if the column stores strings.

Example:

message m {
optional int64 id;
optional int64 date_id;
optional binary datetimestring;
optional int32 dspId;
optional int32 advertiserId;
optional int32 status;
optional int64 bidding_req_num;
optional int64 imp;
optional int64 click_num;
}

No

No default value

nullFormat

The string that represents a null pointer. No standard strings can represent a null pointer in text files. You can use this parameter to define a string that represents a null pointer. For example, if you set nullFormat to null, Data Integration considers null as a null pointer.

No

No default value

header

The headers in the files that you want to write to OSS. Example: ['id', 'name', 'age'].

No

No default value

maxFileSize (advanced parameter, which is available only in the code editor)

The maximum size of a single file that can be written to OSS. Default value: 100,000. Unit: MB. OSS Writer performs object rotation based on the value of this parameter. Object rotation is similar to log rotation of Log4j. When a file is uploaded to OSS in multiple parts, the maximum size of a part is 10 MB. This size is the minimum granularity used for object rotation. If you set this parameter to a value that is less than 10 MB, the maximum size of a single file that can be written to OSS is still 10 MB. The InitiateMultipartUploadRequest operation can be used to upload a file in a maximum of 10,000 parts at a time.

If object rotation occurs, suffixes, such as _1, _2, and _3, are appended to the new object names that consist of prefixes and random UUIDs.

No

100,000

suffix (advanced parameter, which is available only in the code editor)

The file name extension of the files that you want to write to OSS. For example, if you set the suffix parameter to .csv, the final name of a file written to OSS is in the fileName****.csv format.

No

No default value