How to use the scan-based analysis feature of Simple Log Service - Simple Log Service

Simple Log Service's scan-based analysis feature enables log analysis based on specific fields without configuring indexes for those fields. This topic describes the fundamental syntax for scan-based analysis.

Procedure

Upon receiving a scan-based query request, Simple Log Service executes the following steps:

Executes the search statement to retrieve logs.

Important
The search statement requires indexes. For instance, before executing the status:200 | set session mode=scan;SELECT api, count(*) AS PV GROUP BY api statement, you must create an index for the status field, but not for the api field.
Scans the search statement's query results based on the specified fields in the analytic statement and extracts data for these fields.
Executes the analytic statement to aggregate and analyze the extracted data, returning the analysis results.

Basic syntax

To indicate the use of scan mode, prepend set session mode=scan; to standard SQL.

Search statement | set session mode=scan; Analytic statement (standard SQL)

Example

* and status:200 | set session mode=scan;SELECT api, count(*) AS PV GROUP BY api

Limits

In scan-based analysis mode, all fields are treated as varchar type.

To analyze fields of other data types, use the cast function to convert data types. For more information, see the cast function.
For example, to calculate the average latency for each API, use the avg function, which requires a numeric type. First, convert the latency field to bigint using the cast function.
```
status:200 | set session mode=scan;SELECT avg(cast(latency AS bigint)) AS 'Average Latency', api GROUP BY api
```
In scan-based analysis mode, Simple Log Service scans logs to extract data for fields specified in the analytic statement. Existing field indexes are not utilized.
For instance, should you create a field index of the long type for the status field, the index for the status field will not be effective when using the specified statement in scan mode.
```
* | set session mode=scan;SELECT count(1) AS pv, api WHERE cast(status AS bigint) = 200 GROUP BY api
```
In such cases, it is recommended to use the following statement.

Note
If an index exists for a field, it is advisable to place the field before the vertical bar (|) as a filter condition.
```
status:200 | set session mode=scan;SELECT count(1) as pv, api GROUP BY api
```
The volume of data that can be analyzed in scan-based analysis mode is limited. Within a shard, a maximum of 500,000 logs can be analyzed and up to 10 million logs can be scanned. If a search statement is used, the scan limit pertains to the number of logs returned by the search statement. Exceeding these limits results in the system returning partial analysis results and indicating that the analysis is incomplete.
To avoid incomplete results, consider the following recommendations:
- For large-scale structured data analysis, index-based analysis is recommended. Create field indexes and enable the log analysis feature.
- Use search statements to filter logs and minimize the data volume for scanning. Ensure indexes are created for fields used in search statements.
- Narrow the query time range to reduce the data volume for scanning.
The *|select * statement is not supported in scan-based analysis mode.
When conducting JOIN analysis on multiple Logstores in scan-based analysis mode, prefix the Logstore name to the field name in the format LogstoreName.key.

Comparison between index-based analysis and scan-based analysis

Comparison item	Index-based analysis	Scan-based analysis
Syntax	`Search statement \| Analytic statement (standard SQL)`	`Search statement \| set session mode=scan; Analytic statement (standard SQL)`
Need for indexes	Indexes must be configured for fields, and the log analysis feature must be enabled.	Indexes are not required. Important However, the search statement before the vertical bar (\|) does require indexes.
SQL syntax	Supports a broad range of SQL syntax and functions. For more information, see the function overview.	Supports the same SQL syntax and functions as index-based analysis, with some limitations in certain scenarios. For more information, see Limits.
Data analysis	Capable of analyzing hundreds of millions of rows of logs. For more information, see the query and analysis overview. The Dedicated SQL feature can be enabled for enhanced analysis.	Capable of analyzing millions of rows of logs. For large data volumes, it is recommended to filter logs with a search statement that requires indexes.
Fee structure	Charges apply for index traffic and storage. For more information, see Billable items of the pay-as-you-go billing method.	Charges are based on scan traffic, which corresponds to the volume of data scanned. Logs are identified based on index-based query results.