Simple Log Service provides the rule-based consumption feature to consume logs that meet specific conditions in real time. This topic describes the rule-based consumption feature.
The rule-based consumption feature is available in the following regions: China (Beijing), China (Chengdu), China (Guangzhou), Malaysia (Kuala Lumpur), Philippines (Manila), UK (London), and US (Silicon Valley).
Scenarios
Consume data over the Internet with reduced traffic costs
For example, you want to consume logs over the Internet after you upload the logs to Simple Log Service. Then, you want to filter the logs and distribute the logs to the internal system. In this case, you can use the rule-based consumption feature to filter logs in Simple Log Service. This way, a large number of invalid logs are not shipped to consumers. This helps reduce traffic costs.
Compute data in an on-premises machine with reduced computing resources and increased computing speed
For example, you want to consume logs by using an on-premises machine after you upload the logs to Simple Log Service. Then, you want to compute data on the on-premises machine. In this case, you can use the rule-based consumption feature to compute data based on specific rules in Simple Log Service. This helps reduce a large number of computing resources and increase the computing speed.
Basic syntax
*| where bool_expression
Example:
*| where level = '123'
Limits
A query statement must be less than 10 KB in length.
A query statement must start with
*| where
. You can modify the WHERE clause.You cannot use the
__time__
field to filter logs.You cannot specify a JOIN, ORDER BY, GROUP BY, HAVING, or AGGREGATE clause in a query statement.
Subqueries are not supported. You cannot specify a column as the result set.
All fields in a WHERE clause are of the text type. If you want to use a function whose input parameter is not of the text type, such as the abs function, you must use the cast function to convert the data type of the corresponding field. For more information, see Data type conversion functions.
In scenarios in which no data is returned, all data is filtered out in a request. In this case, the value of the nextCursor parameter in the returned result is changed.
How rule-based consumption works
Simple Log Service provides the rule-based consumption feature to match logs based on specific filter rules and ship the matched logs to consumers.
Query parser: parses and verifies query statements and then generates filter rules.
Data filter: scans all raw data to match the data that meets the filter rules.
.
Differences between common data consumption and rule-based consumption
Item | Common data consumption | Rule-based consumption |
Data size | The maximum size of compressed data that can be read after you call the LogGroupList operation is 1 MB. | The maximum size of compressed data that can be read after you call the LogGroupList operation is 1 MB. |
Operators | No | SQL subsets are supported. Data is processed by row. For more information, see Operators. |
Consumption-related quota | If the number of concurrent data consumption jobs exceeds the limit after read and write traffic is throttled, error 403 is reported. | If the number of concurrent data consumption jobs exceeds the limit after read and write traffic is throttled, error 403 is reported. |
Billing
The rule-based consumption feature is free of charge during the public preview. After the feature is officially released, you are charged fees when you use the feature. For more information, see the announcements and updates of Simple Log Service. During the public preview, you are charged for the read and write traffic that is generated when you consume data based on the amount of data that is filtered based on rules. For more information, see Billable items of pay-by-feature.
Operator
The following table describes the operators that are supported by the rule-based consumption feature.
Operator | Description |
= | Matches logs in which the value of a field is equal to a value. Example: |
!= | Matches logs in which the value of a field is not equal to a value. Example: |
> | Matches logs in which the value of a field is greater than a value. Example: |
>= | Matches logs in which the value of a field is greater than or equal to a value. Example: |
< | Matches logs in which the value of a field is less than a value. Example: |
<= | Matches logs in which the value of a field is less than or equal to a value. Example: |
like | Matches a specific character pattern in a string and queries logs in which a field contains a value. Example: |
in | Specifies multiple values in a WHERE clause. Example: |
is null | Matches logs in which the value of a field is null. Example: |
is not null | Matches logs in which the value of a field is not null. Example: |
and | The and operator. Example: |
or | The or operator. Example: |
not | The not operator. Example: |
( ) | This operator is used to increase the priority of the query conditions that are enclosed in parentheses (). Example: |
Supported functions
For more information, see Scan-based query overview.