All Products
Search

This Product

    Simple Log Service:Log topics

    Document Center

    Simple Log Service:Log topics

    Last Updated:Dec 02, 2024

    This topic describes how to configure log topics to distinguish the text logs that are collected from different servers or users.

    Configure log topics

    Prerequisites

    Procedure

    You can configure log topics in the Simple Log Service console or by calling API operations.

    Console

    • On the Logtail Configuration page, click Other Global Configurations and select a log topic type from the Log Topic Type drop-down list. The following list describes the values of the Log Topic Type parameter.

      • Machine Group Topic: If you select this type, you must configure the Machine Group Topic parameter when you create a machine group. For more information, see Create a machine group.

      • File Path Extraction: If you select this type, you must specify a regular expression.

      • Custom: If you select this type, you must specify customized:// + Custom topic name to use static log topics.

    • Create a Logtail configuration in the Simple Log Service console. For more information, see Collect text logs from servers.

    API

    You can call the CreateConfig operation to create a Logtail configuration. For more information about the parameters related to log topics, see the topicFormat parameter in Logtail configurations.

    Log topic types

    • Machine Group Topic

      Simple Log Service allows you to apply one Logtail configuration to multiple machine groups. This type is used to distinguish the logs that are generated by servers in different machine groups. When Logtail uploads the logs of servers in different machine groups to Simple Log Service, Logtail includes the topics specified by the Machine Group Topic parameter in logs as log topics. You can use the log topics to query the logs.

    • File Path Extraction

      This type is used to distinguish the logs that are generated for different users or by different applications. If the logs of different users or applications are stored in different root directories but the directories contain subdirectories or files that have duplicate names, Logtail cannot identify the user or application for or by which specific logs are generated when Logtail collects the logs.

      You can specify a regular expression in a Logtail configuration to match a complete log file path. In this case, Logtail includes the names of the users or applications that are found in match results in logs as log topics when Logtail uploads the logs to Simple Log Service. You can use the log topics to query the logs.

      Note

      You must escape the forward slashes (/) in a regular expression that is used to match a log file path.

      Scenario 1: The logs that are generated for different users are stored in different root directories. However, the files of the logs are the same. Example: 

      /data/logs
├── userA
│   └── serviceA
│       └── service.log
├── userB
│   └── serviceA
│       └── service.log
└── userC
    └── serviceA
        └── service.log

      If you specify /data/logs as the log file path and specify service.log as the log file name in a Logtail configuration, Logtail collects logs from all files named service.log to a Logstore. The user for which a specific log is generated cannot be identified. You can specify the following regular expression to extract a value from each log file path. Each value is used as a unique log topic.

      • Regular expression

        \/data\/logs\/(.*)\/serviceA\/.*

      • Extraction results

        __topic__: userA
__topic__: userB
__topic__: userC

      Scenario 2: If the source of a log cannot be identified by using a single log topic, you can specify multiple capturing groups in the regular expression that is used to match a log file path to extract key information. Capturing groups include named capturing groups and unnamed capturing groups. A named capturing group is in the (?P<name>) format. If the capturing groups in the specified regular expression are all named capturing groups, tag fields are generated in the __tag__:{name} format. If the capturing groups in the specified regular expression are all unnamed capturing groups, tag fields are generated in the __tag__:__topic_{i}__ format. {i} indicates the ordinal number of an unnamed capturing group.

      Note

      If a regular expression includes multiple capturing groups, the __topic__ field is not generated.

      For example, the log file path is /data/logs/userA/serviceA/service.log. You can specify one of the following regular expressions to extract multiple values from the log file path.

      • Example 1: Extract multiple values by using unnamed capturing groups in a regular expression.

        • Regular expression

          \/data\/logs\/(.*?)\/(.*?)\/service.log

        • Extraction results

          __tag__:__topic_1__: userA
__tag__:__topic_2__: serviceA

      • Example 2: Extract multiple values by using named capturing groups in a regular expression.

        • Regular expression

          \/data\/logs\/(?P<user>.*?)\/(?P<service>.*?)\/service.log

        • Extraction results

          __tag__:user: userA
__tag__:service: serviceA

    • Custom

      You can specify customized:// + Custom topic name to use static log topics.

      Important

      Custom log topics are supported only by Logtail V0.16.21 and later. In this case, Logtail must be installed on a Linux server.

    Log query by log topic

    On the query and analysis page of a Logstore, specify __topic__: userA and __tag__:__topic_1__: userA to query logs by log topic. For more information, see Search syntax.