Simple Log Service:Collect text logs in regex mode

Parameter

Description

Configuration Name

Enter a name for the Logtail configuration. The name must be unique in a project. After you create the Logtail configuration, you cannot change its name.

Log Topic Type

Select a method to generate log topics. For more information, see Log topics.

• Machine Group Topic: The topics of the machine groups are used as log topics. If you want to distinguish the logs from different machine groups, select this option.

• File Path Extraction: You must specify a custom regular expression. A part of the file path that matches the regular expression is used as the log topic. If you want to distinguish the logs from different sources, select this option.

• Custom: You must specify a custom log topic.

Advanced Parameters

Optional. Configure the advanced parameters that are related to global configurations. For more information, see CreateLogtailPipelineConfig.

Parameter

Description

File Path

Specify the directory and name of log files based on the location of the logs on your server, such as an ECS instance.

• If you specify a file path in a Linux operating system, the path must start with a forward slash (/). Example: /apsara/nuwa/**/app.Log.

• If you specify a file path in a Windows operating system, the path must start with a drive letter. Example: C:\Program Files\Intel\**\*.Log.

You can specify an exact directory and an exact name. You can also use wildcard characters to specify the directory and name. For more information, see Wildcard matching. When you configure this parameter, you can use only the asterisk (*) or question mark (?) as wildcard characters.

Simple Log Service scans all levels of the specified directory to find the log files that match the specified conditions. Examples:

• If you specify /apsara/nuwa/**/*.log, Simple Log Service collects logs from the log files whose names are suffixed by .log in the /apsara/nuwa directory and the recursive subdirectories of the directory.

• If you specify /var/logs/app_*/**/*.log, Simple Log Service collects logs from the log files that meet the following conditions: The file name is suffixed by .log. The file is stored in a subdirectory of the /var/logs directory or in a recursive subdirectory of the subdirectory. The name of the subdirectory matches the app_* pattern.

• If you specify /var/log/nginx/**/access*, Simple Log Service collects logs from the log files whose names start with access in the /var/log/nginx directory and the recursive subdirectories of the directory.

Maximum Directory Monitoring Depth

Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of subdirectories that can be matched by the ** wildcard characters included in the value of File Path. A value of 0 indicates that only the log file directory that you specify is monitored.

File Encoding

Select the encoding format of log files.

First Collection Size

Specify the size of data that Logtail can collect from a log file the first time Logtail collects logs from the file. The default value of First Collection Size is 1024. Unit: KB.

• If the file size is less than 1,024 KB, Logtail collects data from the beginning of the file.

• If the file size is greater than 1,024 KB, Logtail collects the last 1,024 KB of data in the file.

You can configure First Collection Size based on your business requirements. Valid values: 0 to 10485760. Unit: KB.

Collection Blacklist

If you turn on Collection Blacklist, you must configure a blacklist to specify the directories or files that you want Simple Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only the asterisk (*) or question mark (?) as wildcard characters.

The following types of blacklists are supported: File Path Blacklist, File Blacklist, and Directory Blacklist.

Allow File to Be Collected for Multiple Times

By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs from a log file, you must turn on Allow File to Be Collected for Multiple Times.

Advanced Parameters

Optional. Configure the advanced parameters that are related to input plug-ins. For more information, see CreateLogtailPipelineConfig.

Parameter

Description

Log Sample

Add a sample log that is collected from an actual scenario. You can use the sample log to configure parameters that are related to log processing with ease. You can add multiple sample logs. Make sure that the total length of the logs does not exceed 1,500 characters.

[2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
    at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
    at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
    at TestPrintStackTrace.main(TestPrintStackTrace.java:16)

Multi-line Mode

• Specify the type of multi-line logs. A multi-line log spans multiple consecutive lines. You can configure this parameter to identify each multi-line log in a log file.

  • Custom: A multi-line log is identified based on the value of Regex to Match First Line.

  • Multi-line JSON: Each JSON object is expanded into multiple lines. Example:

    {
      "name": "John Doe",
      "age": 30,
      "address": {
        "city": "New York",
        "country": "USA"
      }
    }

• Configure Processing Method If Splitting Fails.

  Exception in thread "main" java.lang.NullPointerException
      at com.example.MyClass.methodA(MyClass.java:12)
      at com.example.MyClass.methodB(MyClass.java:34)
      at com.example.MyClass.main(MyClass.java:½0)

  For the preceding sample log, Simple Log Service can discard the log or retain each single line as a log when it fails to split the log.

  • Discard: The log is discarded.

  • Retain Single Line: Each line of log text is retained as a log. Four logs are retained.

Processing Method

Select Processors. You can add native plug-ins and extended plug-ins for data processing. For more information about Logtail plug-ins for data processing, see Logtail plug-ins overview.

• Logtail earlier than V2.0

  • You cannot add native plug-ins and extended plug-ins at the same time.

  • You can use native plug-ins only to collect text logs. When you add native plug-ins, take note of the following items:

    • You must add one of the following Logtail plug-ins for data processing as the first plug-in: Data Parsing (Regex Mode), Data Parsing (Delimiter Mode), Data Parsing (JSON Mode), Data Parsing (NGINX Mode), Data Parsing (Apache Mode), and Data Parsing (IIS Mode).

    • After you add the first plug-in, you can add one Time Parsing plug-in, one Data Filtering plug-in, and multiple Data Masking plug-ins.

  • You can add extended plug-ins only after you add native plug-ins.

• Logtail V2.0

  • You can arbitrarily combine native plug-ins for data processing.

  • You can combine native plug-ins and extended plug-ins. Make sure that extended plug-ins are added after native plug-ins.