Filter logs - Simple Log Service - Alibaba Cloud Documentation Center

If you use Logtail to collect logs, you can use the processor_filter_regex or processor_filter_key_regex plug-in to filter logs. This topic describes the parameters of the processor_filter_regex and processor_filter_key_regex plug-ins. This topic also provides examples on how to configure the plug-ins.

Limits

The input plug-ins for text logs and container stdout and stderr support only form configuration. Other input plug-ins support only editor configuration in JSON.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview of Logtail plug-ins for data processing.

processor_filter_regex

You can use regular expressions to match the values of log fields and filter logs.

Form configuration

Parameters

Set Processor Type to Filter Logs with Regular Expression (Match Log Field Values). Then, configure other parameters based on the following table.

Note

A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.

Parameter

Description

Collect Logs

Specifies the name of a log field and the regular expression based on which the value of the field is matched. If the values of log fields match the specified regular expressions, the log is collected.

You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical AND.

Drop Log

Specifies the name of a log field and the regular expression based on which the value of the field is matched. If the value of a log field matches a specified regular expression, the log is dropped.

You can add multiple key-value pairs. The key-value pairs are evaluated by using a logical OR.

Configuration example

In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.

Raw logs

Log 1

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "chrome"

Log 3

"ip" : "192.168.*.*"
"method" : "POST"
"browser" : "ali-sls-ilogtail"

Logtail plug-in configuration for data processing

Results

Log	Collected	Reason
Log 1	No	The value of the browser field matches the regular expression that is specified in the Drop Log parameter.
Log 2	Yes	The values of the specified fields match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expression that is specified in the Drop Log parameter.
Log 3	No	The value of the ip field does not match a regular expression that is specified in the Collect Logs parameter.

Editor configuration in JSON

Parameters

Set type to processor_filter_regex. Then, configure other parameters in detail based on the following table.

Note

A log is collected only if the values of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.

Parameter	Type	Required	Description
Include	JSON Object	No	The key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical AND. If the values of log fields match the regular expressions that are specified in the Include parameter, the log is collected.
Exclude	JSON Object	No	The key-value pairs that are used as match conditions. In a key-value pair, the key specifies a field, and the value specifies a regular expression based on which the value of the field is matched. The key-value pairs are evaluated by using a logical OR. If the value of a log field matches a regular expression that is specified in the Exclude parameter, the log is dropped.

Configuration example

In this example, only logs whose ip starts with 10, method is POST, and browser is not aliyun.* are collected.

Raw logs

Log 1

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"ip" : "10.**.**.**"
"method" : "POST"
"browser" : "chrome"

Log 3

"ip" : "192.168.*.*"
"method" : "POST"
"browser" : "ali-sls-ilogtail"

Logtail plug-in configuration for data processing

{
   "type" : "processor_filter_regex",
    "detail" : {
         "Include" : {
            "ip" : "10\..*",
            "method" : "POST"
         },
         "Exclude" : {
            "browser" : "aliyun.*"
         }
    }
}

Results

Log	Collected	Reason
Log 1	No	The value of the browser field matches the regular expression that is specified in the Exclude parameter.
Log 2	Yes	The values of the specified fields match the regular expressions that are specified in the Include parameter and do not match the regular expression that is specified in the Exclude parameter.
Log 3	No	The value of the ip field does not match a regular expression that is specified in the Include parameter.

processor_filter_key_regex

You can use regular expressions to match the names of log fields and filter logs.

Form configuration

Parameters

Set Processor Type to Filter Logs with Regular Expression (Match Log Field Names). Then, configure other parameters based on the following table.

Note

A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Collect Logs parameter and do not match the regular expressions that are specified in the Drop Log parameter. In other cases, the log is dropped.

Parameter	Description
Collect Logs	Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected.
Drop Log	Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped.

Configuration example

In this example, logs whose field names start with request are collected.

Raw logs

Log 1

"request_time" : 20
"request_length": 2314
"request_method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"request_time" : 70
"request_method" : "GET"
"ip" : "192.168.**.**"

Log 3

"browser" : "ali-sls-ilogtail"
"status" : 200

Logtail plug-in configuration for data processing

Results

Log	Collected	Reason
Log 1	Yes	The names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Collect Logs parameter.
Log 2	Yes	The names of the request_time and request_length fields match the regular expression that is specified in the Collect Logs parameter.
Log 3	No	No field names in the log match the regular expression that is specified in the Collect Logs parameter.

Editor configuration in JSON

Parameters

Set type to processor_filter_key_regex. Then, configure other parameters in detail based on the following table.

Note

A log is collected only if the names of fields in the log exactly match the regular expressions that are specified in the Include parameter and do not match the regular expressions that are specified in the Exclude parameter. In other cases, the log is dropped.

Parameter	Type	Required	Description
Include	string[]	No	Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical AND. If the names of log fields match the specified regular expressions, the log is collected.
Exclude	string[]	No	Specifies the regular expression based on which the name of a log field is matched. Multiple regular expressions are evaluated by using a logical OR. If the name of a log field matches a specified regular expression, the log is dropped.

Configuration example

In this example, logs whose field names start with request are collected.

Raw logs

Log 1

"request_time" : 20
"request_length": 2314
"request_method" : "POST"
"browser" : "aliyun-sdk-java"

Log 2

"request_time" : 70
"request_method" : "GET"
"ip" : "192.168.**.**"

Log 3

"browser" : "ali-sls-ilogtail"
"status" : 200

Logtail plug-in configuration for data processing

{
   "type" : "processor_filter_key_regex",
    "detail" : {
         "Include" : [
            "request.*"
         ]
    }
}

Results

Log	Collected	Reason
Log 1	Yes	The names of the request_time, request_length, and request_method fields match the regular expression that is specified in the Include parameter.
Log 2	Yes	The names of the request_time and request_length fields match the regular expression that is specified in the Include parameter.
Log 3	No	No field names in the log match the regular expression that is specified in the Include parameter.