Search syntax and examples - Simple Log Service - Alibaba Cloud Documentation Center

A Simple Log Service project allows you to execute statements to query logs that are stored in a Logstore. The query results can be independently used or used in analytic statements for complex data analysis and processing. This topic describes the writing process and syntax of a query statement.

Statement writing process

You can perform the following steps to write a query statement:

Select a search type.
Select a field data type.
Select a match mode.

Note

For more information about the raw logs of the query examples in this topic, see Debugging.

Select a search type

Important

To query logs in a Logstore, you must first create an index. If you want to analyze a specific field, you must create field indexes and use a SELECT statement. For more information about how to create field indexes and full-text indexes, see Create indexes.
Query and analysis results vary based on index configurations. If you create full-text indexes and field indexes, the field indexes take precedence.

A Simple Log Service project supports full-text searches and field-specific searches based on the types of indexes that you create. For more information, see Create indexes. The following list describes the differences between the full-text search and field-specific search:

By default, if you enable full-text indexing, the data types of all fields in a log except the __time__ field are set to text. For more information, see Full-text indexes.
If you enable field indexing, the data types of the related fields can be text, JSON, long, and double. For more information, see Field indexes.
If you create only full-text indexes for your Logstore, you can use only full-text search.
If you create field indexes for your Logstore, you can query logs based on the field types:
- double and long: You can use only field-specific search syntax to specify query conditions.
- text: If you can identify the field with which a keyword is associated and field indexes are created for the field, we recommend that you use field-specific search syntax to specify query conditions. If you cannot identify the field with which the keyword is associated, we recommend that you use the full-text search syntax.
  - If you use the full-text search syntax but full-text indexes are not created for your Logstore, you can query data only from fields whose indexed data is of the text type.
  - If you use the full-text search syntax and full-text indexes are created for your Logstore, you can query data from all fields. In this case, the indexed data of all fields is of the text type.

Full-text search

You cannot query data from a specific field based on full-text search syntax. Full-text search syntax: keywords1 [ [ and | or | not ] keywords2 ] ...

keywords1 specifies the keyword that you want to use to query data. You can also use asterisks (*) and question marks (?) to perform fuzzy match. To combine query conditions, you can use operators such as and and or.

Example 1: Query logs that contain the GET keyword. Search syntax: GET

Example 2: Query logs that contain the GET or POST keyword. Search syntax: GET or POST.

Field-specific search

You can query data from specific fields based on field-specific search syntax. The fields can be of the text, long, double, and JSON type. Field-specific search syntax:

indexname1 [ : | > | >= | < | <= | = | in ] keyword1 [ [ and | or | not ] indexname2 ... ]

Important

indexname1 specifies the name of the field from which you want to query data. If a proper noun such as a field name or a table name contains special characters such as spaces and Chinese characters or syntax keywords such as and and or, you must enclose the characters or keywords in double quotation marks (""). For more information, see How do I use quotation marks in query statements?
If indexes are created for fields of the long or double type, you can use the following comparison operators: >, >=, <, <=, =, and in.

Example 1: Query logs whose request_method field value is GET. Search syntax: request_method: GET.

Example 2: Query logs whose request_time_msec field value is greater than 50. Search syntax: request_time_msec>50. The index type of the field is double.

Example 3: Query logs whose request_method field value is GET and request_time_msec field value is greater than 50. Search syntax: request_method: GET and request_time_msec>50.

Select a field data type

When you write a search statement, take note of the characteristics of field data types and use the required operators. This way, you can use the statement to obtain logs in an efficient and accurate manner.

Field data types

Field data type	Description	Supported operator
Text	You can query data of the string type by setting the data types of related fields to text. By default, if you enable full-text indexing, the data types of all fields in a log except the `__time__` field are set to text.	`and`, `or`, `not`, `()`, `:`, `""`, `\`, `*`, and `?`.
Long and double	You can query the value of a field by using a numeric range only if you set the data type of the field to long or double. If you do not set the data type of a field to double or long or the syntax of the numeric range is invalid, Simple Log Service performs a full-text search, and the search result may be different from the expected result. For example, if you execute the `owner_id>100` search statement and the data type of the `owner_id` field is not double or long, logs that contain `owner_id`, `>`, and `100` are returned. In this example, the greater-than sign (>) is not a delimiter. If you change the data type of a field from text to double or long, you can use only the equal-to sign (`=`) to query data. If you want to use ranges and comparison operators such as the greater-than (>) and less-than (<) signs to query data, you must reindex the data. For more information, see Reindex logs for a Logstore.	`and`, `or`, `not`, `()`, `>`. `>=`, `<`, `<=`, `=`, and `in`.
JSON	You can set the data type of a field in JSON objects to long, double, or text based on the field value, and turn on Enable Analytics for the field.	Specify operators based on the data types of the fields in JSON objects.

Operator

Important

The letters of the in operator must be in lowercase. Other operators are not case-sensitive.
Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").
The following operators are listed in descending order of priority:
1. Colons (:)
2. Double quotation marks ("")
3. Parentheses ()
4. and and not
5. or

Operator	Description
`:`	This operator is used for field-specific searches based on the key:value format. Example: `request_method:GET`. If a field name or a field value contains special characters such as spaces, colons (:), and hyphens (-), you must enclose the field name or field value in double quotation marks (""). Example: `"file info":apsara`.
`and`	The `and` operator. Example: `request_method:GET and status:200`. If no syntax keywords exist among multiple search keywords, the search keywords are evaluated by using the `and` operator. For example, `GET 200 cn-shanghai` is equivalent to `GET and 200 and cn-shanghai`.
`or`	The `or` operator. Example: `request_method:GET or status:200`.
`not`	The `not` operator. Example: `request_method:GET not status:200` and `not status:200`.
`( )`	This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: `(request_method:GET or request_method:POST) and status:200`.
`""`	To convert the keyword to an ordinary character, you can use double quotation marks (`""`) to enclose a syntax keyword. In a field-specific search, the words in the double quotation marks (`""`) are considered as a whole. If a field name or a field value contains special characters such as spaces, Chinese characters, colons (`:`), and hyphens (`-`), or contains syntax keywords such as `and` and `or`, you must enclose the field name or field value in double quotation marks (`""`). For example, `"and"` returns log entries that contain and. In this case, and is not an operator. Simple Log Service supports the following operators: `sort`, `asc`, `desc`, `group by`, `avg`, `sum`, `min`, `max`, and `limit`. If you want to use these operators as keywords, you must enclose the operators in double quotation marks (`""`). If a log is processed by using the data transformation feature or a Logtail plug-in, the key in a tag field is converted to a common key. If you want to search for the log, you must enclose the name of the key in double quotation marks (`""`). Example: `"__tag__:__client_ip__":192.0.2.1`. The `__tag__:__client_ip__` field is a reserved field of Simple Log Service. The field indicates the IP address of the host from which the log is collected. For more information, see Reserved fields.
`\`	The escape character. This character is used to escape double quotation marks (`""`). Escaped double quotation marks ("") indicate double quotation marks (""). For example, if the content of a log is `instance_id:nginx"01"`, you can escape the log content to `instance_id:nginx\"01\"`.
`*`	The wildcard character. This character is used to match zero, one, or multiple characters. Example: `host:wwwcom`. Note* Simple Log Service searches all logs and obtains up to 100 words that match the specified conditions. Then, Simple Log Service returns the logs that contain one or more of these words and match the query conditions.
`?`	The wildcard character. This character is used to match a single character. Example: `host:aliyund?c`.
`>`	This operator is used to query the logs in which the value of a field is greater than a specified numeric value. Example: `request_time>100`.
`>=`	This operator is used to query the logs in which the value of a field is greater than or equal to a specified numeric value. Example: `request_time>=100`.
`<`	This operator is used to query the logs in which the value of a field is less than a specified numeric value. Example: `request_time<100`.
`<=`	This operator is used to query the logs in which the value of a field is less than or equal to a specified numeric value. Example: `request_time<=100`.
`=`	This operator is used to query the logs in which the value of a field is equal to a specified numeric value. Equal-signs (`=`) and colons (`:`) have the same effect on fields of the double or long type. For example, `request_time=100` is equivalent to `request_time:100`.
`in`	This operator is used to query the logs in which the value of a field is within a specified numeric range. Brackets [] indicate a closed interval, and parentheses () indicate an open interval. A space is used to separate two numbers in a numeric range. Example: `request_time in [100 200]` or `request_time in (100 200]`. Important The letters of the in operator must be in lowercase.
`__source__`	This operator is used to query the logs of a specified log source. Wildcard characters are supported. Example: `__source__:192.0.2.`. Important* The __source__ field is a reserved field in Simple Log Service. This field can be abbreviated to source. If you configure a custom source field, the custom field conflicts with the reserved source field in Simple Log Service. If you want to search for the custom field, you must use Source or SOURCE in the search statement.
`__tag__`	This operator is used to query logs by using metadata. Example: `__tag__:__receive_time__:1609837139`.
`__topic__`	This operator is used to query the logs of a specified log topic. Example: `__topic__:nginx_access_log`.

Select a match mode

You can use exact searches or fuzzy searches based on keywords and your business requirements. If the required logs are not returned, troubleshoot the issue based on the instructions in What do I do if no results are returned when I query a log?

Search type

Description

Example

Exact search

Complete words are used for searches.

Simple Log Service uses word segmentation to query logs. In an exact search, phrases cannot be completely matched. For example, the abc def search statement returns the logs that contain abc and def. The phrase abc def cannot be completely matched. If you want the abc def phrase to be completely matched, you can perform a phrase search or use a LIKE clause. For more information, see Phrase search and How do I query logs by using exact match?

host:example.com: returns the logs in which the value of the host field contains example.com.
PUT and cn-shanghai: returns the logs that contain the PUT and cn-shanghai keywords.
* | Select * where http_user_agent like '%like Gecko%': returns the logs in which the value of the http_user_agent field contains the like Gecko phrase.
#"redo_index/1": returns the logs that contain the redo_index/1 phrase.

Fuzzy search

You can add an asterisk (*) or a question mark (?) as a wildcard character to the middle or end of a word in a search statement when you perform a fuzzy search. The word must be 1 to 64 characters in length. If a word contains a wildcard character, Simple Log Service searches all logs and obtains up to 100 words that match the word. Then, Simple Log Service returns the logs that contain one or more of these words. If you specify more accurate words, the search result is more accurate.

Important

You cannot add an asterisk (*) or a question mark (?) to the start of a word.
The long and double data types do not support asterisks (*) or question marks (?) in fuzzy searches. You can specify a numeric range to perform a fuzzy search. Example: status in [200 299].

A fuzzy search is performed based on samples by using the following mechanism:

If you enable the field indexing feature and specify a field to query logs, Simple Log Service obtains random samples from the indexed data of the field and returns results. Simple Log Service does not perform full-text scans.
If you enable the full-text indexing feature and do not specify a field to query logs, Simple Log Service obtains random samples from the indexed data of all fields and returns results. Simple Log Service does not perform full-text scans.

request_time>60 and request_method:Ge*: returns the logs in which the value of the request_time field is greater than 60 and the value of the request_method field starts with Ge.
addr*: obtains 100 words that start with addr from all logs and returns the logs that contain one or more of these words.
host:www.yl*: obtains 100 words that start with www.yl from the values of the host field in all logs and returns the logs that contain one or more of these words.

For more information, see How do I query logs by using fuzzy match?

Examples of search statements

If you execute a search statement on different logs based on different index configurations, the statement returns different results. The examples provided in this section are based on the following sample log and index configurations.

Sample log

An NGINX access log is used as the sample log.

日志样例

Index configurations

Before you can execute a search statement, make sure that indexes are configured. For more information, see Create indexes.

On the query and analysis page of your Logstore, click Index Attributes in the upper-right corner and select Attributes.

Check whether indexes are configured. The following figure shows fields for which indexes are configured.

Common search examples

Expected search result	Search statement	Debugging
Logs that record successful GET requests (status codes: 200 to 299)	`request_method:GET and status in [200 299]`	Debugging
Logs that record GET requests and in which the source region of the requests is not the China (Hangzhou) region	`request_method:GET not region:cn-hangzhou`	None
Logs that record GET requests or POST requests	`request_method:GET or request_method:POST`	Debugging
Logs that do not record GET requests	`not request_method:GET`	Debugging
Logs that record successful GET requests or successful POST requests	`(request_method:GET or request_method:POST) and status in [200 299]`	Debugging
Logs that record failed GET requests or failed POST requests	`(request_method:GET or request_method:POST) not status in [200 299]`	Debugging
Logs that record successful GET requests (status codes: 200 to 299) and in which the request duration is less than 60 seconds	`request_method:GET and status in [200 299] not request_time>=60`	Debugging
Logs in which the request duration is equal to 60 seconds	`request_time:60`	Debugging
Logs in which the request duration is equal to 60 seconds	`request_time=60`	Debugging
Logs in which the request duration is greater than or equal to 60 seconds and is less than 200 seconds	`request_time>=60 and request_time<200`	Debugging
	`request_time in [60 200)`	Debugging
Logs in which the request_time field is empty or the value of the field is an invalid number	`request_time:* not request_time > -10000000000` Note `request_time:` is used to check whether the `request_time` field exists. In this search statement, the asterisk () does not indicate a fuzzy search.	Debugging
Logs that contain the request_time field and in which the value of the field is a number	`request_time > -1000000000`	Debugging
Logs that contain and	`"and"` Note In this search statement, and is a common string but not an operator.	Debugging
Logs in which the value of the request method field is PUT	`"request method":PUT` Important The name of the request method field contains spaces. You must enclose the field name in double quotation marks ("") in a search statement.	None
Logs whose topic is HTTPS or HTTP	`__topic__:HTTPS or __topic__:HTTP`	None
Logs that are collected from the 192.0.2.1 host	`__tag__:__client_ip__:192.0.2.1` The `__tag__:__client_ip__` field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields. Important If a log is processed by using the data transformation feature or a Logtail plug-in, the key in the __tag__:__client_ip__ field is converted to a common key. If you want to search for the log, you must enclose the name of the __tag__:__client_ip__ field in double quotation marks ("") in the search statement. Example: `"__tag__:__client_ip__":192.0.2.1`.	None
Logs whose IP addresses match `192.168.XX.XX`	`* \| select * from log where key like '192.168.%.%'` For more information, see Use the LIKE clause to implement fuzzy match.	None
Logs in which the remote_user field is not empty	`not remote_user:""`	Debugging
Logs in which the remote_user field is empty	`remote_user:""`	Debugging
Logs in which the value of the remote_user field is not null	`not remote_user:"null"`	Debugging
Logs that do not contain the remote_user field	`not remote_user:*`	Debugging
Logs that contain the remote_user field	`remote_user:*`	Debugging
Logs in which the value of the city field is not Shanghai	`not city:Shanghai` Note If you want to query Chinese strings, you must turn on Include Chinese when you configure indexes. For more information, see Create indexes.	None

Advanced search examples

Fuzzy searches

Expected search result	Search statement	Debugging
Logs that contain specific words. The words start with cn.	`cn*`	Debugging
Logs in which the value of the region field starts with cn.	`region:cn*`	None
Logs in which the value of the region field contains cn*.	`region:"cn"` Note* In this search statement, `cn` is a complete string. Examples: If the content of a log is `region:cn,en` and the delimiter is a comma (,), Simple Log Service splits the log content into `region`, `cn`, and `en`. You can use the search statement to search for the log. If the content of a log is `region:cnhangzhou`, Simple Log Service considers `cn*hangzhou` as a whole. In this case, you cannot use the search statement to search for the log.	None
Logs that contain specific words. The words start with mozi, end with la, and include one character between mozi and la.	`mozi?la`	Debugging
Logs that contain specific words. The words start with mo, end with la, and include zero, one, or more characters between mo and la.	`mo*la`	Debugging
Logs that contain specific words. The words start with moz or sa.	`moz* and sa*`	Debugging
Logs in which the value of the region field ends with hai.	You cannot use a search statement to search for the logs. You can use the LIKE clause in an SQL statement to search for the logs. For more information, see Use the LIKE clause to implement fuzzy match. `* \| select * from log where region like '%hai'`	None

Delimiter-based searches

Simple Log Service splits the content of a log into multiple words based on the delimiters that you specify. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If you leave Delimiter empty, Simple Log Service considers the value of each field as a whole. In this case, you can search for a log only by using a complete string or performing a fuzzy search. For more information about how to specify delimiters, see Create indexes.

For example, the value of the http_user_agent field is Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/192.0.2.0 Safari/537.2.

If you leave Delimiter empty, Simple Log Service considers the field value as a whole. In this case, you cannot search for logs by using the http_user_agent:Chrome search statement.
If you set Delimiter to , '";=()[]{}?@&<>/:\n\t\r, Simple Log Service splits the field value into Mozilla, 5.0, Windows, NT, 6.1, AppleWebKit, 537.2, KHTML, like, Gecko, Chrome, 192.0.2.0, Safari, and 537.2. In this case, you can search for logs by using the http_user_agent:Chrome search statement.

Important

If a search keyword contains delimiters, you can perform a phrase search or use a LIKE clause. Examples:

Phrase search: #"redo_index/1". For more information, see Phrase search.
LIKE clause: * | select * from log where key like 'redo_index/1'.

Expected search result	Search statement	Debugging
Logs in which the value of the http_user_agent field contains Chrome	`http_user_agent:Chrome`	Debugging
Logs in which the value of the http_user_agent field contains Linux and Chrome	`http_user_agent:Linux and http_user_agent:Chrome`	Debugging
	`http_user_agent:"Linux Chrome"`	Debugging
Logs in which the value of the http_user_agent field contains Firefox or Chrome	`http_user_agent:Firefox or http_user_agent:Chrome`	Debugging
Logs in which the value of the request_uri field contains /request/path-2	`request_uri:/request/path-2`	Debugging
Logs in which the value of the request_uri field starts with /request and does not contain /file-0	`request_uri:/request* not request_uri:/file-0`	Debugging
Logs in which the `redo_index/1` phrase is completely matched	`#"redo_index/1"` `* \| select * from log where key like 'redo_index/1'` Note You can perform a phrase search or use a LIKE clause to completely match the phrase. If you perform an exact search, words such as `redo_index` and `1` are matched.	None

Query examples in special scenarios

Search statements

To convert the keyword to an ordinary character, you can use double quotation marks ("") to enclose a syntax keyword. In a field-specific search, the words in the double quotation marks ("") are considered as a whole.

If a field name or a field value contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-), or contains syntax keywords such as and and or, you must enclose the field name or field value in double quotation marks (""). For example, "and" returns log entries that contain and. In this case, and is not an operator.
Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").
If a log is processed by using the data transformation feature or a Logtail plug-in, the key in a tag field is converted to a common key. If you want to search for the log, you must enclose the name of the key in double quotation marks (""). Example: "__tag__:__client_ip__":192.0.2.1. The __tag__:__client_ip__ field is a reserved field of Simple Log Service. The field indicates the IP address of the host from which the log is collected. For more information, see Reserved fields.

Expected search result	Search statement
Logs in which the value of the `request method` field contains `PUT`. The name of the `request method` field contains a space.	`"request method":PUT`
Logs that are collected from the `192.0.2.1` host.	`"__tag__:__client_ip__":192.0.2.1`

Analytic statements

If a proper noun such as a field name or a table name contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-) or contains syntax keywords such as and and or, you must enclose the proper noun in double quotation marks ("") in analytic statements.
If specific characters represent a string, you must use single quotation marks ('') to enclose the characters in analytic statements. Strings that are not enclosed or enclosed in double quotation marks ("") indicate field names or column names. For example, 'status' indicates the status string, and status or "status" indicates the status log field.

Expected search result	Search statement
Logs whose IP addresses match `192.168.XX.XX`	`* \| select * from log where key like '192.168.%.%'`

For more information, see How do I use quotation marks in query statements?

References

How do I use quotation marks in query statements?
For more information about analytic functions and syntax, see Analytic functions and syntax.
Optimize query performance
Troubleshooting
- FAQ about log query
- How do I resolve common errors that may occur when I query and analyze logs?
Log query examples
- How do I precisely sort query logs by time?
- Query and analyze website logs
- Analyze Layer 7 access logs of SLB
- You can query JSON logs whose field values are JSON objects or JSON arrays. For more information about query and analysis examples, see Query and analyze JSON logs and FAQ about the query and analysis of JSON logs.