Related information about the scan-based query feature - Simple Log Service

Simple Log Service provides the scan-based query feature that scans logs based on specified fields and returns query results. The feature allows you to query logs without the need to configure indexes for the logs. The feature also allows you to use Simple Log Service Processing Language (SPL) statements to filter, convert, and parse query results. This topic describes the scan-based query feature.

Background information

Simple Log Service provides the index-based query feature. However, in some scenarios, indexes cannot be created or used. Examples:

Indexes are not created for some fields to reduce costs. In this scenario, an urgent need to query historical logs may not be met.
The length of a field value exceeds the upper limit supported by indexes. In this scenario, the excess part cannot be queried.
A large number of field names are involved and cannot be obtained in advance. In this scenario, field indexes cannot be created.
The type of a field is changed. In this scenario, the field cannot be queried.

To address the preceding issues, Simple Log Service provides the scan-based query and scan-based analysis features. You can use the features to query and analyze logs without the need to create indexes for the fields that are used for scanning. For more information about the scan-based analysis feature, see Scan-based analysis overview.

Note

The scan-based query feature supports SPL. For more information, see SPL overview. You can use SPL statements to extract structured information from the raw data that is obtained, process fields in the raw data, and filter the raw data. SPL statements support multi-level data processing.

Benefits

Low costs: You do not need to create indexes. You are not charged for index traffic or index storage.
Flexible implementation: You can specify field types based on your business requirements regardless of whether indexes are created or index types are specified.
Optimized search capability: More functions are supported.

Limits

The execution of SPL statements for scan-based query is limited. For more information, see Limits.
Random page turning is not supported.

How it works

When Simple Log Service receives a scan-based query request, Simple Log Service performs the following steps:

Executes the search statement to query logs.
Important
The search statement requires indexes. If index-based query is not required, you can also specify an asterisk (*) as the search statement. For example, before the status:200 | WHERE userId = '123' | extend host=upper(hostname) statement is executed, you must create an index for the status field. You do not need to create indexes for the userId and hostname fields.
Executes SPL statements on the query results of the search statement and then returns the final results. For example, you can execute an SPL statement to filter, convert, and parse data.

Basic syntax

Syntax
For more information about the syntax of SPL instructions, see SPL instructions.
```
Index-based search statement | <spl-cmd> ... | <spl-cmd> ...
```

Examples

Filter raw logs based on a specific condition.
```
status:200 | where host like '%www%'
```

Generate a new field and filter data based on the new field.

status:200 
| extend timediff = cast(endTime as bigint) - cast(beginTime as bigint)
| where timediff > 100

Expand a JSON field to obtain new fields, and discard the original JSON field.
```
status:200 
| parse-json body 
| project-away body
```

Comparison between index-based query and scan-based query

Item	Index-based query	Scan-based query
Syntax	`Search statement`. For more information, see Search syntax.	`Search statement \| SPL instruction 1 \| SPL instruction 2 \| ...`
Whether indexes need to be configured	Yes.	No. Important The index-based search statement requires indexes.
Support for an analytic statement	Supported.	Supported.
Support for random page turning	Supported.	Not supported. Only continuous page turning is supported. You can move forward or backward only one page.
Log histogram	The log histogram is displayed based on the results of the search statement.	The log histogram is displayed based on the results of the search statement and the scan progress.
Operators and functions	Logical calculation and mathematical calculation are supported. Fuzzy search is supported. SQL functions are not supported.	For more information, see SPL instructions and SPL-supported SQL functions.
Field types	Field types are determined by the data types that are specified in index configurations. For more information, see Data types.	The system considers the types of fields in SPL statements as text regardless of whether indexes are configured for the fields. For more information, see Convert data types.
Result size	The number of logs to return can be specified in the Simple Log Service console or by calling an SDK. The maximum number is 100.	If one of the following conditions is met, the system stops the current scan and returns results: The number of identified logs reaches the specified number of logs to return. You can specify the number of logs to return in the Simple Log Service console or by calling an SDK. The number of scanned logs exceeds the upper limit that is automatically specified for the current scan. The default value is 100000, which is determined based on the results of a search statement. The scan duration exceeds 45 seconds.
Fees	You are charged for index traffic and index storage. For more information, see Billable items of pay-by-feature.	You are charged for scans based on the scan traffic, which is equivalent to the amount of data returned after scanning. The system identifies logs based on the results of index-based query.