All Products
Search

This Product

    Simple Log Service:Search syntax

    Document Center

    Simple Log Service:Search syntax

    Last Updated:Dec 23, 2024

    Simple Log Service allows you to execute search statements to query logs that are stored in a Logstore. The search results can be independently used or used in analytic statements for complex data analysis and processing. This topic describes the syntax, scenarios, and examples of search statements.

    Basic syntax

    Each query statement consists of a search statement and an analytic statement. The search statement and the analytic statement are separated with a vertical bar (|). Format:

    Search statement|Analytic statement

    A search statement can be independently executed. An analytic statement must be executed together with a search statement. The log analysis feature is used to analyze data in the search results or all data in a Logstore.

    Important

    • We recommend that you specify up to 30 search conditions in a search statement.

    • You do not need to specify the FROM or WHERE clause in an analytic statement. By default, all data of the current Logstore is analyzed. Analytic statements do not support offsets and are not case-sensitive. You do not need to append a semicolon (;) to an analytic statement.

    Statement

    Description

    Search statement

    A search statement specifies one or more search conditions. A search statement can be a keyword, a numeric value, a numeric value range, a space, or an asterisk (*).

    If you specify a space or an asterisk (*) as the search statement, no conditions are used for searching and all logs are returned.

    Analytic statement

    An analytic statement is used to aggregate or analyze data in the search results or all data in a Logstore. For more information about the functions and syntax supported by Simple Log Service for analyzing logs, see the following topics:

    Sample query statement:

    * | SELECT status, count(*) AS PV GROUP BY status
    Note

    For more information about the raw logs in the search examples of this topic, see Debugging.

    Writing process of search statements

    image

    To write a query statement, perform the following steps:

    Step 1: Select a search type

    Important

    • Before you query logs in a Logstore, you must create an index. If you want to analyze a specific field, you must create field indexes and use a SELECT statement. For more information about how to create field indexes and full-text indexes, see Create indexes.

    • Query and analysis results vary based on index configurations. If you create full-text indexes and field indexes, the field indexes take precedence.

    Searches are classified into full-text searches and field-specific searches based on the index type. The following list describes the differences between full-text searches and field-specific searches:

    • If you create only full-text indexes for your Logstore, you can use only the full-text search syntax to specify query conditions.

    • If you create field indexes for your Logstore, the search syntax that you can use varies based on the data types of fields.

      • double and long: You can use only the field-specific search syntax to specify query conditions.

      • text: If you can identify the field with which a keyword is associated and field indexes are created for the field, we recommend that you use the field-specific search syntax to specify query conditions. If you cannot identify the field with which the keyword is associated, we recommend that you use the full-text search syntax.

        • If you use the full-text search syntax but full-text indexes are not created for your Logstore, you can query data only from fields whose indexed data is of the text type.

        • If you use the full-text search syntax and full-text indexes are created for your Logstore, you can query data from all fields. In this case, the indexed data of all fields is of the text type.

    Full-text search

    You cannot query data from a specific field by using the full-text search syntax. Full-text search syntax: keywords1 [ [ and | or | not ] keywords2 ] ....

    keywords1 specifies the keyword that you want to use to query data. You can also use asterisks (*) and question marks (?) to perform fuzzy match. To combine query conditions, you can use operators such as and and or.

    • Example 1

      Query logs that contain the GET keyword. Search syntax: GET.

    • Example 2

      Query logs that contain the GET or POST keyword. Search syntax: GET or POST.

    Field-specific search

    You can query data from specific fields by using the field-specific search syntax. The fields can be of the text, long, double, or json type. Field-specific search syntax:

    indexname1 [ : | > | >= | < | <= | = | in ] keyword1 [ [ and | or | not ] indexname2 ... ]

    Important

    • indexname1 specifies the name of the field from which you want to query data. If a proper noun such as a field name or table name contains special characters such as spaces and Chinese characters or syntax keywords such as and and or, you must enclose the proper noun in double quotation marks (""). For more information, see How do I use quotation marks in query statements?

    • If indexes are created for fields of the long or double type, you can use the following comparison operators: >, >=, <, <=, =, and in.

    • Example 1

      Query logs whose request_method field value is GET. Search syntax: request_method: GET.

    • Example 2

      Query logs whose request_time_msec field value is greater than 50. Search syntax: request_time_msec>50. The request_time_msec field is of the double type.

    • Example 3

      Query logs whose request_method field value is GET and request_time_msec field value is greater than 50. Search syntax: request_method: GET and request_time_msec>50.

    Step 2: Select a field data type

    When you write a search statement, take note of the characteristics of field data types and use the required operators. This way, you can use the statement to obtain logs in an efficient and accurate manner.

    Field data types

    Field data type

    Description

    Supported operator

    text

    You can query data of the string type by setting the data types of related fields to text. By default, if you enable full-text indexing, the data types of all fields in a log except the __time__ field are set to text.

    and, or, not, (), :, "", \, *, and ?.

    long or double

    You can query the value of a field by using a numeric range only if you set the data type of the field to long or double.

    • If you do not set the data type of a field to double or long or the syntax of the numeric range is invalid, Simple Log Service performs a full-text search and the search result may be different from the expected result.

      For example, if you execute the owner_id>100 search statement and the data type of the owner_id field is not double or long, logs that contain owner_id, >, and 100 are returned. In this example, the greater-than sign (>) is not a delimiter.

    • If you change the data type of a field from text to double or long, you can use only the equal-to sign (=) to query data. If you want to use ranges and comparison operators such as the greater-than (>) and less-than (<) signs to query data, you must reindex the data. For more information, see Reindex logs for a Logstore.

    and, or, not, (), >. >=, <, <=, =, and in.

    json

    You can set the data type of a field in JSON objects to long, double, or text based on the field value, and turn on Enable Analytics for the field.

    Specify operators based on the data types of the fields in JSON objects.

    Operators

    Important

    • The letters of the in operator must be in lowercase. Other operators are not case-sensitive.

    • Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").

    • The following operators are listed in descending order of priority:

      1. Colons (:)

      2. Double quotation marks ("")

      3. Parentheses ()

      4. and and not

      5. or

    Operator

    Description

    :

    This operator is used for field-specific searches based on the key:value format. Example: request_method:GET.

    If a field name or field value contains special characters such as spaces, colons (:), and hyphens (-), you must enclose the field name or field value in double quotation marks (""). Example: "file info":apsara.

    and

    The and operator. Example: request_method:GET and status:200.

    If no syntax keywords exist among multiple search keywords, the search keywords are evaluated by using the and operator. For example, GET 200 cn-shanghai is equivalent to GET and 200 and cn-shanghai.

    or

    The or operator. Example: request_method:GET or status:200.

    not

    The not operator. Example: request_method:GET not status:200 or not status:200.

    ( )

    This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: (request_method:GET or request_method:POST) and status:200.

    ""

    This operator is used to enclose a syntax keyword. If a syntax keyword is enclosed in double quotation marks (""), the keyword is converted to an ordinary word. In a field-specific search, the words that are enclosed in double quotation marks ("") are considered as a whole.

    • If a field name or field value contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-) or syntax keywords such as and and or, you must enclose the field name or field value in double quotation marks (""). For example, "and" returns the logs that contain the word and. In this case, and is not an operator.

    • Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").

    • If a log is processed by using the data transformation feature or a Logtail plug-in, the key in the __tag__:__client_ip__ field is converted to a common key. If you want to search for the log, you must enclose the name of the __tag__:__client_ip__ field in double quotation marks ("") in the search statement. Example: "__tag__:__client_ip__":192.0.2.1. The __tag__:__client_ip__ field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields.

    \

    The escape character. This character is used to escape double quotation marks (""). Double quotation marks ("") can indicate themselves only after they are escaped. For example, if the content of a log is instance_id:nginx"01", you can execute the instance_id:nginx\"01\" statement to search for the log.

    *

    The wildcard character, which is used to match zero or more characters. Example: host:www*com.

    Note

    Simple Log Service searches all logs to query 100 words that match the specified conditions. Then, Simple Log Service returns the logs that contain one or more of the words that match the query conditions.

    ?

    The wildcard character, which is used to match a single character. Example: host:aliyund?c.

    >

    This operator is used to query the logs in which the value of a field is greater than a specific numeric value. Example: request_time>100.

    >=

    This operator is used to query the logs in which the value of a field is greater than or equal to a specific numeric value. Example: request_time>=100.

    <

    This operator is used to query the logs in which the value of a field is less than a specific numeric value. Example: request_time<100.

    <=

    This operator is used to query the logs in which the value of a field is less than or equal to a specific numeric value. Example: request_time<=100.

    =

    This operator is used to query the logs in which the value of a field is equal to a specific numeric value. Equal-signs (=) and colons (:) have the same effect on fields of the double or long type. For example, request_time=100 is equivalent to request_time:100.

    in

    This operator is used to query the logs in which the value of a field is within a specific numeric range. Brackets [] indicate a closed interval, and parentheses () indicate an open interval. A space is used to separate two numbers in a numeric range. Example: request_time in [100 200] or request_time in (100 200].

    Important

    The letters of the in operator must be in lowercase.

    __source__

    This operator is used to query the logs of a specific log source. Wildcard characters are supported. Example: __source__:192.0.2.*.

    Important

    The __source__ field is a reserved field in Simple Log Service. This field can be abbreviated to source. If you configure a custom source field, the custom field conflicts with the reserved source field in Simple Log Service. If you want to search for the custom field, you must use Source or SOURCE in the search statement.

    __tag__

    This operator is used to query logs by using metadata. Example: __tag__:__receive_time__:1609837139.

    __topic__

    This operator is used to query the logs of a specific log topic. Example: __topic__:nginx_access_log.

    Step 3: Select a match mode

    You can use exact searches or fuzzy searches based on keywords and your business requirements. If the required logs are not returned, troubleshoot the issue based on the instructions in What do I do if no results are returned when I query a log?

    Search type

    Description

    Example

    Exact search

    Complete words are used for searches.

    Simple Log Service uses word segmentation to query logs. In an exact search, phrases cannot be completely matched. For example, the abc def search statement returns the logs that contain abc and def. The phrase abc def cannot be completely matched. If you want the abc def phrase to be completely matched, you can perform a phrase search or use a LIKE clause. For more information, see Phrase search and How do I query logs by using exact match?

    • host:example.com: returns the logs in which the host field value contains example.com.

    • PUT and cn-shanghai: returns the logs that contain the PUT and cn-shanghai keywords.

    • * | Select * where http_user_agent like '%like Gecko%': returns the logs in which the http_user_agent field value contains the like Gecko phrase.

    • #"redo_index/1": returns the logs that contain the redo_index/1 phrase.

    Fuzzy search

    You can add an asterisk (*) or a question mark (?) as a wildcard character to the middle or end of a word in a search statement when you perform a fuzzy search. The word must be 1 to 64 characters in length. If a word contains a wildcard character, Simple Log Service searches all logs to query 100 words that match the word. Then, Simple Log Service returns the logs that contain one or more of the words. If you specify more accurate words, the search result is more accurate.

    Important

    • You cannot add an asterisk (*) or a question mark (?) to the start of a word.

    • The long and double data types do not support asterisks (*) or question marks (?) in fuzzy searches. You can specify a numeric range to perform a fuzzy search. Example: status in [200 299].

    A fuzzy search is a sample query that uses the following mechanism:

    • If you enable the field indexing feature and specify a field to query logs, Simple Log Service obtains random samples from the indexed data of the field and returns results. Simple Log Service does not perform full-text scans.

    • If you enable the full-text indexing feature and do not specify a field to query logs, Simple Log Service obtains random samples from the indexed data of all fields and returns results. Simple Log Service does not perform full-text scans.

    • request_time>60 and request_method:Ge*: returns the logs in which the request_time field value is greater than 60 and the request_method field value starts with Ge.

    • addr*: queries 100 words that start with addr in all logs and returns the logs that contain one or more of the words.

    • host:www.yl*: queries 100 words that start with www.yl in the host field values of all logs and returns the logs that contain one or more of the words.

    For more information, see How do I query logs by using fuzzy match?

    Examples of search statements

    If you execute a search statement on different logs based on different index configurations, the statement returns different results. The examples provided in this section are based on the following sample log and index configurations.

    Sample log

    An NGINX access log is used as the sample log.

    日志样例

    Index configurations

    Before you execute a search statement, make sure that indexes are configured. For more information, see Create indexes. To check index configurations, perform the following steps:

    1. On the query and analysis page of a Logstore, choose Index Attributes > Attributes.image

    2. In the Search & Analysis panel, check whether field indexes are configured.索引

    Common search examples

    Expected search result

    Search statement

    Debugging

    Logs that record successful GET requests (status codes: 200 to 299) 

    request_method:GET and status in [200 299]

    Debugging

    Logs that record GET requests and in which the source region of the requests is not the China (Hangzhou) region 

    request_method:GET not region:cn-hangzhou

    None

    Logs that record GET requests or POST requests 

    request_method:GET or request_method:POST

    Debugging

    Logs that do not record GET requests 

    not request_method:GET

    Debugging

    Logs that record successful GET or POST requests 

    (request_method:GET or request_method:POST) and status in [200 299]

    Debugging

    Logs that record failed GET or POST requests 

    (request_method:GET or request_method:POST) not status in [200 299]

    Debugging

    Logs that record successful GET requests (status codes: 200 to 299) and in which the request duration is less than 60 seconds 

    request_method:GET and status in [200 299] not request_time>=60

    Debugging

    Logs in which the request duration is equal to 60 seconds 

    request_time:60

    Debugging

    request_time=60

    Debugging

    Logs in which the request duration is greater than or equal to 60 seconds and is less than 200 seconds 

    request_time>=60 and request_time<200

    Debugging

    request_time in [60 200)

    Debugging

    Logs in which the request_time field is empty or the field value is an invalid number 

    request_time:* not request_time > -10000000000
    Note

    request_time:* is used to check whether the request_time field exists. In this search statement, the asterisk (*) does not indicate a fuzzy search.

    Debugging

    Logs that contain the request_time field and in which the field value is a number 

    request_time > -1000000000

    Debugging

    Logs that contain and 

    "and"
    Note

    In this search statement, and is a common string but not an operator.

    Debugging

    Logs in which the request method field value is PUT 

    "request method":PUT
    Important

    The name of the request method field contains spaces. You must enclose the field name in double quotation marks ("") in a search statement.

    None

    Logs whose topic is HTTPS or HTTP 

    __topic__:HTTPS or __topic__:HTTP

    None

    Logs that are collected from the 192.0.2.1 host 

    __tag__:__client_ip__:192.0.2.1

    The __tag__:__client_ip__ field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields.

    Important

    If a log is processed by using the data transformation feature or a Logtail plug-in, the key in the __tag__:__client_ip__ field is converted to a common key. If you want to search for the log, you must enclose the name of the __tag__:__client_ip__ field in double quotation marks ("") in the search statement. Example: "__tag__:__client_ip__":192.0.2.1.

    None

    Logs whose IP addresses match 192.168.XX.XX 

    * | select * from log where key like '192.168.%.%'

    For more information, see Use the LIKE clause to implement fuzzy match.

    None

    Logs in which the remote_user field is not empty 

    not remote_user:""

    Debugging

    Logs in which the remote_user field is empty 

    remote_user:""

    Debugging

    Logs in which the remote_user field value is not null 

    not remote_user:"null"

    Debugging

    Logs that do not contain the remote_user field 

    not remote_user:*

    Debugging

    Logs that contain the remote_user field 

    remote_user:*

    Debugging

    Logs in which the city field value is not Shanghai 

    not city:Shanghai
    Note

    If you want to query Chinese strings, you must turn on Include Chinese when you configure indexes. For more information, see Create indexes.

    None

    Advanced search examples

    • Fuzzy searches

      Expected search result

      Search statement

      Debugging

      Logs that contain specific words. The words start with cn. 

      cn*

      Debugging

      Logs in which the region field value starts with cn. 

      region:cn*

      None

      Logs in which the region field value contains cn*. 

      region:"cn*"
      Note

      In this search statement, cn* is a complete string. Examples:

      • If the content of a log is region:cn*,en and the delimiter is a comma (,), Simple Log Service splits the log content into region, cn*, and en. You can use the search statement to search for the log.

      • If the content of a log is region:cn*hangzhou, Simple Log Service considers cn*hangzhou as a whole. In this case, you cannot use the search statement to search for the log.

      None

      Logs that contain specific words. The words start with mozi, end with la, and include one character between mozi and la. 

      mozi?la

      Debugging

      Logs that contain specific words. The words start with mo, end with la, and include zero or more characters between mo and la. 

      mo*la

      Debugging

      Logs that contain specific words. The words start with moz or sa. 

      moz* and sa*

      Debugging

      Logs in which the region field value ends with hai.

      You cannot use a search statement to search for the logs. You can use the LIKE clause in an SQL statement to search for the logs. For more information, see Use the LIKE clause to implement fuzzy match. 

      * | select * from log where region like '%hai'

      None

    • Delimiter-based searches

      Simple Log Service splits the content of a log into multiple words based on the delimiters that you specify. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If you leave Delimiter empty, Simple Log Service considers the value of each field as a whole. In this case, you can search for a log only by using a complete string or performing a fuzzy search. For more information about how to specify delimiters, see Create indexes.

      For example, the http_user_agent field value is Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/192.0.2.0 Safari/537.2.

      • If you leave Delimiter empty, Simple Log Service considers the field value as a whole. In this case, you cannot search for logs by using the http_user_agent:Chrome search statement.

      • If you set Delimiter to , '";=()[]{}?@&<>/:\n\t\r, Simple Log Service splits the field value into Mozilla, 5.0, Windows, NT, 6.1, AppleWebKit, 537.2, KHTML, like, Gecko, Chrome, 192.0.2.0, Safari, and 537.2. In this case, you can search for logs by using the http_user_agent:Chrome search statement.

      Important

      If a search keyword contains delimiters, you can perform a phrase search or use a LIKE clause. Examples:

      • Phrase search: #"redo_index/1". For more information, see Phrase search.

      • LIKE clause: * | select * from log where key like 'redo_index/1'.

      Expected search result

      Search statement

      Debugging

      Logs in which the http_user_agent field value contains Chrome 

      http_user_agent:Chrome

      Debugging

      Logs in which the http_user_agent field value contains Linux and Chrome 

      http_user_agent:Linux and http_user_agent:Chrome

      Debugging

      http_user_agent:"Linux Chrome"

      Debugging

      Logs in which the http_user_agent field value contains Firefox or Chrome 

      http_user_agent:Firefox or http_user_agent:Chrome

      Debugging

      Logs in which the request_uri field value contains /request/path-2 

      request_uri:/request/path-2

      Debugging

      Logs in which the request_uri field value starts with /request and does not contain /file-0 

      request_uri:/request* not request_uri:/file-0

      Debugging

      Logs in which the redo_index/1 phrase is completely matched

      • #"redo_index/1"

      • * | select * from log where key like 'redo_index/1'

      Note

      You can perform a phrase search or use a LIKE clause to completely match the phrase. If you perform an exact search, words such as redo_index and 1 are matched.

      None

    Query examples in special scenarios

    • Search statements

      This operator is used to enclose a syntax keyword. If a syntax keyword is enclosed in double quotation marks (""), the keyword is converted to an ordinary word. In a field-specific search, the words that are enclosed in double quotation marks ("") are considered as a whole.

      • If a field name or field value contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-) or syntax keywords such as and and or, you must enclose the field name or field value in double quotation marks (""). For example, "and" returns the logs that contain the word and. In this case, and is not an operator.

      • Simple Log Service supports the following operators: sort, asc, desc, group by, avg, sum, min, max, and limit. If you want to use these operators as keywords, you must enclose the operators in double quotation marks ("").

      • If a log is processed by using the data transformation feature or a Logtail plug-in, the key in the __tag__:__client_ip__ field is converted to a common key. If you want to search for the log, you must enclose the name of the __tag__:__client_ip__ field in double quotation marks ("") in the search statement. Example: "__tag__:__client_ip__":192.0.2.1. The __tag__:__client_ip__ field is a reserved field in Simple Log Service. The field indicates the IP address of the host from which logs are collected. For more information, see Reserved fields.

      Expected query result

      Search statement

      Logs in which the request method field value contains PUT. The name of the request method field contains a space. 

      "request method":PUT

      Logs that are collected from the 192.0.2.1 host. 

      "__tag__:__client_ip__":192.0.2.1

    • Analytic statements

      • If a proper noun such as a field name or table name contains special characters such as spaces, Chinese characters, colons (:), and hyphens (-) or syntax keywords such as and and or, you must enclose the proper noun in double quotation marks ("") in analytic statements.

      • If specific characters represent a string, you must use single quotation marks ('') to enclose the characters in analytic statements. For example, 'status' indicates the status string, and status or "status" indicates the status log field.

      Expected query result

      Query statement

      Logs whose IP addresses match 192.168.XX.XX 

      * | select * from log where key like '192.168.%.%'

      For more information, see How do I use quotation marks in query statements?

    References