Simple Log Service allows you to query and analyze logs in index mode. This topic explains the fundamental concepts of indexes, queries, and analysis.
Index
To retrieve specific content from raw logs, we often use keywords, such as logs containing curl: curl/7.74.0. Without splitting, the log text is treated as a single block, which may not exactly match the keyword curl, resulting in a failed retrieval by Simple Log Service.
For effective retrieval, we can split the log into separate and searchable words using delimiters. Delimiters mark the points where a log is divided. For example, using delimiters such as \n\t\r,;[]{}()&^*#@~=<>/\?:'" will split the log into words such as curl and 7.74.0. Simple Log Service then creates indexes based on these split keywords, enabling you to query and analyze logs more efficiently.
Simple Log Service projects support creating both full-text indexes and field indexes. When both are created, field index configurations are given priority.
Full-text index
A full-text index splits the entire log into multiple text-type words based on specified delimiters. After a full-text index is established, you can perform keyword-based log queries. For example, the query statement Chrome or Safari will return logs containing either Chrome or Safari.
Delimiters do not support Chinese characters. If the Include Chinese option is enabled, Simple Log Service automatically segments Chinese text.
If only full-text indexes are configured, you will be limited to full-text search capabilities. For more information, see Query syntax and functions.
Field index
A field index organizes logs by field names (KEY) and then segments the fields using delimiters. Field indexes support four data types: text, long, double, and JSON. For more information, see Data types. With field indexes, you can refine your queries by specifying field names and values (Key:Value) or use SELECT statements for more advanced searches. For more information, see Field-specific search.
Field indexes are required for querying or analyzing fields (SELECT statements) and take precedence over full-text indexes. If both index types are created, field index configurations take priority.
Fields of the text type can be searched using full-text, field, and analytic (SELECT) queries.
Without full-text indexing, full-text search statements will query all text type fields.
With full-text indexing enabled, full-text search statements will query all logs.
You can query and analyze fields of the long and double types using field search and analytic statements (SELECT).
Query and analysis
Simple Log Service enables log queries at the billion to trillion scale and facilitates statistical analysis of query results using SQL. Query statements can be executed independently, whereas analytic statements must be paired with a query statement, meaning analysis is conducted on either the query results or the entire dataset. For more information, see Query and analyze logs.
Query overview
A query statement defines one or more conditions and retrieves logs that match these criteria. The format is:
query statement, such asstatus: 200.Query conditions can include keywords, numbers, number ranges, spaces, and
*. A space or*indicates no filter condition. For more information, see Query syntax and functions.
Analysis overview
Query and analytic statements are separated by
|. The format is:query statement|analytic statement, for example:* | SELECT status, count(*) AS PV GROUP BY status.To utilize the analytics feature, logs must be collected in a standard logstore and the Enable Statistics option must be activated for the relevant fields during index configuration.
Analytic statements carry out computations and statistical analyses on either the query results or the full dataset. For a comprehensive list of analytic functions and syntax supported by Simple Log Service, see the following topics:
SQL functions: Commonly used for data calculations, transformations, and formatting, such as summing, averaging, string operations, and date processing.
SQL syntax: Used to construct complete SQL queries or data manipulation statements, specifying data sources, conditions, grouping, sorting, and more.
Nested subqueries: Used to embed SELECT statements within other SELECT statements for complex analysis scenarios.
Join query analysis of logstore and MySQL: Performs join queries between logstore and MySQL databases, and saves query results to the MySQL database.
References
Create indexes: To query and analyze logs in a logstore, you must create indexes.
Reindex logs for a logstore: The reindex feature is used to configure or modify indexes for historical data.
Query syntax and functions: Details the basic and advanced features of query statements.
SQL syntax and functions: Outlines common functions and clauses in SQL analytic statements.
Use SPL to query and analyze logs: Simple Log Service Processing Language (SPL) is used to extract structured information, conduct field operations, and filter data from log data.
Machine learning syntax and functions: Apply analytic statements and machine learning functions to analyze field characteristics over time using machine learning algorithms within log query analysis.
Associate Simple Log Service with external data sources: These external data sources include MySQL databases, Alibaba Cloud Object Storage Service (OSS), and hosted CSV files.
Compatibility between Simple Log Service and Elasticsearch: Simple Log Service offers Elasticsearch-compatible APIs. After you collect logs to Simple Log Service, you can use Elasticsearch to query them.