Data types

Data type overview

When you create indexes, Simple Log Service provides the following data types for fields. You can select a data type based on your business requirements.

• Text, long, and double types:

  • If you want to query and analyze fields of the string type, you must set the data type of the fields to text when you create indexes and turn on Enable Analytics for the fields.

  • You can query the value of a field by using a numeric range only if you set the data type of the field to long or double. If you set the data type of a field to long or double, you cannot query the field by using an asterisk (*) or a question mark (?). The wildcard characters are used to perform a fuzzy match.

    • If the value of a field is an integer, set the data type of the field to long.

    • If the value of a field is a floating-point number, set the data type of the field to double. Otherwise, you cannot query the field.

    • If you set the data type of a field to long or double but the value of the field is a string, you cannot query the field.

    Important

    If the value of a field is an invalid numeric value, you can query the field by using the not key > -1000000 search statement. The statement returns the logs in which the value of the field is an invalid numeric value. You can replace -1000000 with a valid numeric value that is less than or equal to the smallest valid numeric value of the field in your logs.

• JSON type: If the value of a field is of the JSON type, you can set the data type of the field to JSON when you create indexes.

  • You can set the data type of a field in JSON objects to long, double, or text based on the field value and turn on Enable Analytics for the field. After you turn on Enable Analytics, Simple Log Service allows you to query and analyze the field.

    If you select Automatically Index All Text Fields in JSON Field, indexes are automatically created for all fields of the text type in JSON objects. After indexes are created, index traffic is generated.

  • For partially valid JSON-formatted data, Simple Log Service can parse only the valid part of the data.

    The following example shows an incomplete JSON log. Simple Log Service can parse the content.remote_addr, content.request.request_length, and content.request.request_method fields.

    content: {
         remote_addr:"192.0.2.0"
         request: {
                 request_length:"73"
                 request_method:"GE

    Important

    • You can create indexes for leaf nodes in JSON objects. You cannot create indexes for child nodes that contain leaf nodes.

    • You cannot create indexes for fields whose values are JSON arrays or fields in a JSON array.

    • If the value of a field is of the Boolean type, you can set the data type of the field to text when you create indexes.

    • A query statement is in the Search statement|Analytic statement format. In an analytic statement, you must enclose a field name in double quotation marks ("") and a string in single quotation marks ('').

  • References

    • For more information about the scenarios and FAQ related to the query and analysis of JSON logs, see FAQ about the query and analysis of JSON logs. The FAQ includes information about how to create indexes, query and analyze indexed JSON fields, use JSON functions, and analyze JSON arrays.

    • For more information about the configurations and procedures related to the query and analysis of JSON logs, see Query and analyze JSON logs.

    • If the data volume of JSON logs that you want to query and analyze is small, you do not need to create indexes for JSON leaf nodes. In this case, you can use JSON functions to query and analyze the logs. In specific scenarios, you can use only JSON functions for query and analysis. For more information about the usage of JSON functions, see When do I need to use JSON functions? For more information about the descriptions and examples of JSON functions, see JSON functions.

Procedure

1. Create indexes in the Simple Log Service console. For more information, see Create indexes.

2. Select data types for fields when you create indexes.

   text, long, and double types

   JSON type

   • Sample log

   • Index configurations

   • Sample query statements

     • To query the logs whose request duration is greater than 60 seconds, execute the following statement: request_time > 60

     • To query the logs whose request duration is greater than or equal to 60 seconds but less than 200 seconds, execute one of the following statements: request_time in [60 200) and request_time >= 60 and request_time < 200.

     • To query the logs whose request status code is 200, execute the following statement: status = 200.

     • To query the logs that do not record GET requests, execute the following statement: not request_method : GET.

     • To query the logs that contain specific words and the words start with cn, execute the following statement: cn*.

     • To collect client distribution statistics, execute the following statement: * | SELECT ip_to_province(client_ip) as province, count(*) AS pv GROUP BY province ORDER BY pv.

   • Sample log

     The following figure provides an example of a JSON log. The log contains the class, latency, status, and info fields and the reserved fields of Simple Log Service. The value of the info field is a JSON object that contains multiple layers.

     

   • Index configurations

     The following list provides more details:

     • The values of the IP and data fields are JSON arrays. Therefore, you cannot create indexes for the IP or data field and you cannot query or analyze data by using the fields.

     • The region and CreateTime fields are in a JSON array. Therefore, you cannot create indexes for the region or CreateTime field and you cannot query or analyze data by using the fields.

   • Sample query statements

     • To query the logs in which the value of the usedTime field is greater than 60 seconds, execute the following statement: info.usedTime > 60.

     • To query the logs in which the value of the success field is true, execute the following statement: info.success : true.

     • To query the logs in which the value of the usedTime field is greater than 60 seconds and the value of the projectName field is not project01, execute the following statement: info.usedTime > 60 not info.param.projectName : project01.

     • To calculate the average duration that is required to obtain the project information, execute the following statement: methodName = getProjectInfo | SELECT avg("info.usedTime") AS avg_time.