Use SPL to query and analyze logs - Simple Log Service - Alibaba Cloud Documentation Center

You can use Simple Log Service Processing Language (SPL) to extract structured data, process fields, and filter data in logs. For more information, see SPL overview. Simple Log Service supports multi-level data processing based on SPL and allows you to export result data after processing.

Background information

If the fields in an SPL statement are indexed and Enable Analytics is turned on for the fields, you can use SPL to query and analyze logs in index mode. If the fields in an SPL statement are not indexed, you can use SPL to query and analyze logs in scan mode. For more information, see Scan-based query overview.

Supported regions

Note

In unsupported regions, the scan mode is used regardless of whether the fields in an SPL statement are indexed.

China (Fuzhou - Local Region) and Indonesia (Jakarta)

Basic syntax

Syntax
For more information about the syntax of SPL instructions, see SPL instructions.
```
Index-based search statement | <spl-cmd> ... | <spl-cmd> ...
```

Examples

Filter raw logs based on a specific condition.
```
status:200 | where host like '%www%'
```

Generate a new field and filter data based on the new field.

status:200 
| extend timediff = cast(endTime as bigint) - cast(beginTime as bigint)
| where timediff > 100

Expand a JSON field to obtain new fields, and discard the original JSON field.
```
status:200 
| parse-json body 
| project-away body
```

Comparison between index-based and scan-based SPL

If the fields in an SPL statement are indexed and Enable Analytics is turned on for the fields, you can use SPL to query and analyze logs in index mode. Otherwise, you can use SPL to query and analyze logs in scan mode.

Item	Index-based SPL	Scan-based SPL
Whether indexes need to be created	Indexes must be created for the fields in the SPL statement, and Enable Analytics must be turned on for the fields.	No. Important The index-based search statement requires indexes.
Performance	High.	Moderate.
Support for random page turning	Supported.	Not supported.
Log histogram	The log histogram is displayed based on the results of the search statement.	The log histogram is displayed based on the results of the search statement and the scan progress.
Operators and functions	For more information, see SPL instructions and SPL-supported SQL functions.	For more information, see SPL instructions and SPL-supported SQL functions.
Field types	The system considers the types of the fields in the SPL statement as text. For more information, see Convert data types.	The system considers the types of the fields in the SPL statement as text. For more information, see Convert data types.
Result size	You can specify the number of logs that you want to return for a query operation in the Simple Log Service console or by calling an SDK. You can specify up to 100 logs.	If one of the following conditions is met, the system stops the current scan and returns results: The number of identified logs reaches the specified number of logs to return. You can specify the number of logs that you want to return for a query operation in the Simple Log Service console or by calling an SDK. The number of scanned logs exceeds the upper limit that is automatically specified for the current scan. By default, the result size of a search statement is 10,0000. The scan duration exceeds 45 seconds.
Fees	You are charged for index traffic and index storage. For more information, see Billable items of pay-by-feature. You are not charged for using SPL.	You are charged for scans based on the scan traffic, which is equivalent to the amount of data returned after scanning. The system identifies logs based on the results of index-based query.

Operation methods

Important

Before you query logs, make sure that logs are collected and indexes are created. An index is a storage structure used to sort one or more columns of log data. For more information, see Create indexes.

Console

Log on to the Simple Log Service console. Go to the query and analysis page of a Logstore and execute a query statement. For more information, see Query and analyze logs.

Example

The number of raw logs is 10 million. The SPL statement is Status:200 | where Category like '%xx%'. A total of 1,000 logs that meet the Status:200 and where Category like '%xx%' conditions are returned. The log histogram on the query and analysis page displays the distribution of the 1,000 logs over time based on the query results.

API

Call the GetLogs or GetLogsV2 operation to query logs. The query results of the GetLogs operation are not compressed and are directly returned. The query results of the GetLogsV2 operation are compressed and then returned.