scan query syntax and features - Simple Log Service - Alibaba Cloud Documentation Center

Simple Log Service's scan query feature enables scanning of target fields without the need for configured indexes, facilitating log queries. It also allows for additional data filtering, transformation, and parsing of query results using SPL statements. This topic describes the fundamental syntax for scan queries.

Workflow

Upon receiving a scan request, Simple Log Service executes the following steps:

Performs a search statement to retrieve logs.
Important
The initial pipeline's search statement relies on index-based queries. Use * if index filtering is not necessary. For instance, to execute status:200 | WHERE userId = '123' | extend host=upper(hostname), an index must be created for the status field, but not for the userId or hostname fields.
Applies SPL statements to the search results and returns the final output. For instance, SPL statements can be used to filter, transform, and parse data.

Basic syntax

The scan query mode is compatible with SPL (SLS Processing Language). For more information, see SPL syntax. Using SPL, you can extract and process structured information from raw data, manipulate fields within that data, and apply data filtering. SPL enables multi-tiered data processing.

Index-based search statement | <spl-cmd> ... | <spl-cmd> ...

Example

Filter raw logs based on a condition
```
status:200 | where host like '%www%'
```

Calculate a new field, and the calculated field can be filtered again.

status:200 
| extend timediff = cast(endTime as bigint) - cast(beginTime as bigint)
| where timediff > 100

Expand JSON fields and discard the original JSON fields.
```
status:200 
| parse-json body 
| project-away body
```

Limits

To understand the runtime limits when executing SPL in scan mode, refer to Limits.
Random page turning is not supported.

Comparison between index-based and scan-based queries

Comparison item	Index-based query	Scan-based query
Syntax	`Search statement` For more information, see query syntax and features.	`Search statement \| SPL instruction 1 \| SPL instruction 2 \| ...` For more information, see the SPL syntax documentation.
Need for configured indexes	Yes.	No. Important Indexes are required for the search statement before the first VERTICAL LINE (\|).
Analytic statement support	Yes.	Yes.
Random page turning support	Yes.	No. Only sequential page turning is supported, allowing for forward or backward navigation by one page at a time.
Log histogram	Displayed based on search statement results.	Displayed based on search statement results and scan progress.
Operators and functions	Supports logical and mathematical operations, fuzzy search, but not SQL functions.	For more information, see the SPL instructions and the list of SQL functions supported by SPL.
Field type	The types of fields are defined by the data types specified in the index configurations. For more information, see Data types.	The system treats all field types in SPL statements as text, irrespective of index configurations for those fields. For more information, see Data type conversion.
Result size	Configurable in the Simple Log Service console or via SDK, with a maximum of 100 logs per query.	The scan halts and returns results if any of the following conditions are met: The identified log count reaches the specified return limit. Configurable in the Simple Log Service console or via SDK. The scanned log count exceeds the preset upper limit of 100,000, based on search statement results. The scan duration exceeds 45 seconds.
Billable items	Charges apply to both index traffic and index storage. For more information, see the details on billable items for the pay-as-you-go billing method.	Charges are based on scan traffic, equivalent to the data volume returned post-scan. Log identification is based on index query results.