SPL - Simple Log Service - Alibaba Cloud Documentation Center

This topic describes Simple Log Service Processing Language (SPL), including its implementation, syntax, and instruction expressions.

How SPL works

Simple Log Service supports SPL in the following features: Logtail collection, ingest processor, scan-based query and log analysis, rule-based data consumption, data transformation (new version), and log query and analysis. The following figure shows how SPL works.

Note

For more information about the capabilities that are supported by SPL in different scenarios, see SPL in different scenarios and SPL cases.

SPL-SPL Doc - ch.jpeg

Limits

Category	Item	Logtail collection	Ingest processor	Real-time consumption	Overview of data transformation (new version)	Scan-based query
SPL complexity	Number of script pipeline levels	16	16	16	64	16
SPL complexity	Script length	64KB	64KB	64KB	128KB	64KB
SPL runtime	Memory size Important For more information, see Handle errors.	50MB	1GB	1GB	1GB	2GB
SPL runtime	Timeout period Important For more information, see Handle errors.	1 second	5 seconds	5 seconds	5 seconds	2 seconds

SPL syntax

SPL statements

An SPL statement supports multi-level data processing, uses a vertical bar (|) as the pipeline symbol for connection, and ends with a semicolon (;). The following section describes the SPL syntax.

Syntax

<data-source> | <spl-expr> | <spl-expr> ;

Parameters

Parameter

Description

data-source

The data source, including a Logstore and an SPL-defined dataset.

Example: * | project status, body.

For more information about the data sources that are supported by SPL in different scenarios, see SPL in different scenarios.

spl-expr

The data processing expression. For more information, see SPL instruction expressions.

Syntax symbols

Symbol	Description
*	The placeholder that allows you to specify a Logstore as the data source of your SPL statement.
.	The keyword of the SPL syntax if an SPL statement starts with a period (.).
\|	The SPL pipeline symbol, which is used to introduce an SPL instruction expression. Format: `\| <spl-cmd> ...`.
;	The end identifier of an SPL statement. This symbol is optional in a single statement or in the last statement among multiple statements.
'...'	The quotes that are used to enclose a string constant.
"..."	The quotes that are used to enclose a field name or a field name pattern.
--	The comment symbol for single-line content.
/.../	The comment symbol for multi-line content.

SPL data types

The following table describes the data types that are supported by SPL.

Category	Data type	Description
Basic numeric data types	BOOLEAN	The BOOLEAN data type.
	TINYINT	An integer with a width of 8 bits.
	SMALLINT	An integer with a width of 16 bits.
	INTEGER	An integer with a width of 32 bits.
	BIGINT	An integer with a width of 64 bits.
	HUGEINT	An integer with a width of 128 bits.
	REAL	A variable-precision floating-point number with a width of 32 bits.
	DOUBLE	A variable-precision floating-point number with a width of 64 bits.
	TIMESTAMP	A UNIX timestamp that is accurate to the nanosecond.
	DATE	The date data type. Format: YYYY-MM-DD.
	VARCHAR	The variable-length character data type.
	VARBINARY	The variable-length binary data type.
Structured numeric data types	ARRAY	The array data type. Brackets (`[]`) are used to access an element in an array based on the corresponding index. The index starts from 1. Example: `* \| extend a = ARRAY[0, 1, 2] \| extend b = a[1]`.
Structured numeric data types	MAP	The dictionary data type. A dictionary key must be of a basic numeric data type. A dictionary value can be of an arbitrary data type. Brackets (`[]`) are used to access an element in a dictionary. Example: `* \| extend a = MAP(ARRAY['foo', 'bar'], ARRAY[0, 1]) \| extend b = a['foo']`.
JSON data type	JSON	The JSON data type.

For more information about how to convert data types during SPL-based data processing, see General references.

SPL instruction expressions

Instruction expression syntax

cmd -option=<option> -option ... <expression>, ... as <output>, ...

Parameters

Parameter	Description
cmd	The name of the instruction.
option	Optional. The execution parameter of the instruction. You must specify a parameter name and start the name with a hyphen `(-)`. You can specify multiple execution parameters in an arbitrary order. The following parameter formats are supported: `-option=<option>:` Specify the parameter in the key-value pair format to execute the instruction. -option: Specify whether to turn on the switch of a feature. By default, all switches are turned off. If you configure this parameter, the switch of a feature is turned on.
expression	Required. The processing logic on the data source. You do not need to specify a parameter name. The position of the <expression> parameter must comply with the definition of the instruction. The following types of expressions are supported: SPL expression An expression that is used to match a field name or a field name pattern, such as content. If you specify characters other than [a-zA-Z_], enclose the characters in double quotation marks (""). Example: `"__tag__:__path__"`. An expression that is used to assign a value to a field, such as `level='INFO'`. You can assign the extraction result of a regular expression to another field as a value. Example: `msg=regex_extract(content, '\S+')`. SQL expression An expression that is used to return the result of data calculation. Example: `cast(status as int)>=500`.
output	The output field that contains the processing result. Example: `\| parse-csv content as a, b, c`.

Instructions

The following table describes the instructions that are supported by SPL.

Category	Instruction	Description
Field processing instructions	project	Retains the fields that match the specified pattern and renames the specified fields.
	project-away	Removes the fields that match the specified pattern and retains all other fields as they are.
	project-rename	Renames the specified fields and retains all other fields as they are.
SQL calculation instructions on structured data	extend	Creates fields based on the result of SQL expression-based data calculation. For more information about the SQL functions that are supported by SPL, see SPL-supported SQL functions.
SQL calculation instructions on structured data	where	Filters data based on the result of SQL expression-based data calculation. For more information about the SQL functions that are supported by SPL, see SPL-supported SQL functions.
Semi-structured data extraction instructions	parse-csv	Extracts information in the CSV format from the specified field.
	parse-json	Extracts the first-layer JSON information from the specified field.
	parse-regexp	Extracts the information that matches groups in the specified regular expression from the specified field.