Container Service for Kubernetes:Collect text logs from Alibaba Cloud ACK clusters (DaemonSet deployment of Logtail)

CRD-AliyunPipelineConfig (recommended)

To create a Logtail collection configuration, simply create the AliyunPipelineConfig custom resource. The resource takes effect automatically after creation.

1. Log on to the ACK console.

2. In the left-side navigation pane, click Cluster List.

3. On the Cluster List page, click Details of the target cluster, and then select the Connection Information tab.

4. Click Manage Cluster Through Workbench in the upper-right corner.

5. Create a new YAML file, modify it according to your actual situation, and use the following sample script:

   This YAML creates a Logtail collection configuration named example-k8s-file and collects the test.LOG file content from all containers in the cluster whose names contain app in multi-line text mode from the path /data/logs/app_1. The logs are sent to a Logstore named k8s-file in a project named k8s-log-test.

   You need to modify the project in the example, such as logging on to Simple Log Service Console, and use the project named k8s-log-<your_cluster_id> generated for you when you installed Logtail in Simple Log Service. Modify the FilePaths in the example to use your container's file path. For more information, see Container File Path Mapping. If you want to customize the content of the YAML file, see CR Parameter Description for detailed information about the parameters of the AliyunPipelineConfig custom resource. If you are interested in the details of the Logtail collection configuration provided by the config item in the YAML file, such as supported input, output, processing plug-in types, and container filtering methods, see PipelineConfig.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   kind: ClusterAliyunPipelineConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster. This name is also the name of the Logtail collection configuration created. If the name is duplicated, it will not take effect.
     name: example-k8s-file
   spec:
     # Specify the target project
     project:
       name: k8s-log-test
     logstores:
     # Create a Logstore named k8s-file
       - name: k8s-file
     # Define the Logtail collection configuration
     config:
       # Sample log (optional)
       sample: |
         2024-06-19 16:35:00 INFO test log
         line-1
         line-2
         end
       # Define input plug-ins
       inputs:
         # Use the input_file plug-in to collect multi-line text logs from containers
         - Type: input_file
           # File path in the container
           FilePaths:
             - /data/logs/app_1/**/test.LOG
           # Enable the container discovery feature.
           EnableContainerDiscovery: true
           # Add conditions to filter containers. Multiple conditions are evaluated by using a logical AND.
           ContainerFilters:
             # Specify the namespace to which the pod that contains the container to be collected belongs. Regular expression matching is supported.
             K8sNamespaceRegex: default
             # Specify the name of the container to be collected. Regular expression matching is supported.
             K8sContainerRegex: ^(.*app.*)$
           # Enable multi-line log collection. Delete this configuration for single-line log collection.
           Multiline:
             # Specify the custom mode to match the beginning of the first line of a log based on a regular expression
             Mode: custom
             # Configure the regular expression for the beginning of a line
             StartPattern: \d+-\d+-\d+.*
       # Define processing plug-ins
       processors:
         # Use the processor_parse_regex_native plug-in to parse logs based on the specified regular expression
         - Type: processor_parse_regex_native
           # Specify the name of the input field
           SourceKey: content
           # Specify the regular expression that is used for the parsing. Use capturing groups to extract fields
           Regex: (\d+-\d+-\d+\s*\d+:\d+:\d+)\s*(\S+)\s*(.*)
           # Specify the fields that you want to extract
           Keys: ["time", "level", "msg"]
       # Define output plug-ins
       flushers:
         # Use the flusher_sls plug-in to send logs to a specific Logstore.
         - Type: flusher_sls
           # Make sure that the Logstore exists
           Logstore: k8s-file
           # Make sure that the endpoint is valid
           Endpoint: cn-hangzhou.log.aliyuncs.com
           Region: cn-hangzhou
           TelemetryType: logs

6. Execute kubectl apply -f example.yaml, where example.yaml is the name of the YAML file you created. Logtail will begin collecting text logs from containers and sending them to Simple Log Service.

CRD-AliyunLogConfig

To create a Logtail collection configuration, simply create the AliyunLogConfig custom resource. The configuration takes effect automatically after creation.

1. Log on to the ACK console.

2. In the left-side navigation pane, click Cluster List.

3. On the Cluster List page, click Details of the target cluster, and then select the Connection Information tab.

4. Click Manage Cluster Through Workbench in the upper-right corner.

5. Create a new YAML file in Workbench, modify it according to your actual situation, and use the following sample script:

   This YAML script creates a Logtail collection configuration named example-k8s-file and collects the test.LOG file content from all containers in the cluster whose names start with app in simple text mode from the path /data/logs/app_1. The logs are sent to a Logstore named k8s-file in a project named k8s-log-<your_cluster_id>.

   You may need to modify the logPath in the example to your actual log file path. If you want to customize the content of the YAML file, see CR Parameter Description for detailed information about the parameters of the AliyunLogConfig custom resource. If you are interested in the details of the Logtail collection configuration provided by the logtailConfig item in the YAML file, such as supported input, output, processing plug-in types, and container filtering methods, see AliyunLogConfigDetail.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster.
     name: example-k8s-file
     # Specify the namespace to which the resource belongs.
     namespace: kube-system
   spec:
     # Specify the name of the project. If you leave this parameter empty, the project named k8s-log-<your_cluster_id> is used.
     # project: k8s-log-test
     # Specify the name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore.
     logstore: k8s-file
     # Specify the Logtail collection configuration.
     logtailConfig:
       # Specify the type of data source. If you want to collect text logs, set the value to file.
       inputType: file
       # Specify the name of the Logtail collection configuration. The name must be the same as metadata.name.
       configName: example-k8s-file
       inputDetail:
         # Specify the simple mode to collect text logs.
         logType: common_reg_log
         # Specify the path to the log file.
         logPath: /data/logs/app_1
         # Specify the log file name. You can use wildcard characters such as asterisks (*) and question marks (?). Example: log_*.log.
         filePattern: test.LOG
         # If you want to collect container text logs, you must set dockerFile to true.
         dockerFile: true
         # Enable multi-line log collection. Delete this configuration for single-line log collection.
         # Regular expression for the beginning of a line. This regular expression indicates the beginning of a log line.
         logBeginRegex: \d+-\d+-\d+.*
         # Specify the conditions that are used to filter containers.
         advanced:
           k8s:
             K8sPodRegex: '^(app.*)$'

6. Execute kubectl apply -f example.yaml, where example.yaml is the name of the YAML file you created. Logtail will begin collecting text logs from containers and sending them to Simple Log Service.

Simple Log Service Console

1. Log on to the Simple Log Service Console.

2. Select the project that you used when you installed the Logtail component from the Project list, such as k8s-log-<your_cluster_id>. On the project page, click the target Logstore's Logtail Configuration, add a Logtail configuration, and click Kubernetes-file Access Now.

3. On the Machine Group Configuration page, select the k8s-group-${your_k8s_cluster_id} machine group under the ACK Daemonset method in the K8s scenario, and click > to add it to the Application Machine Group, and then click Next.

4. Create a Logtail collection configuration. Fill in the required configurations as described below, and then click Next. The Logtail collection configuration takes effect in about 1 minute. Please be patient.

   This section describes only the main configurations. For detailed configurations, see Logtail Collection Configuration.

   ```html

   • Global Configuration

     Enter the configuration name in Global Configuration.

     

   • Input Configuration

     • Logtail Deployment Mode: Select DaemonSet.

     • File Path Type: Choose whether the file path to be collected is a path in the container or on the host. If a hostPath volume is mounted to a container and you want to collect logs from files based on the mapped file path on the container host, set this parameter to Host Path. Otherwise, set it to Path in Container.

     • File Path: This denotes the directory for log collection, which must begin with a forward slash (/). For instance, /data/wwwlogs/main/**/*.Log refers to all files ending in .Log within the /data/wwwlogs/main directory. To specify the maximum depth for directory monitoring—how deep the wildcard ** can go in the File Path—adjust the maximum directory monitoring depth value. A value of 0 means only the current directory is under surveillance.

5. Create Index and Preview Data: Simple Log Service automatically enables full-text indexing by default, indexing all log fields for queries. Additionally, you can manually create field indexes from the collected logs or click Auto Generate Index. This feature allows Simple Log Service to generate field indexes, enabling precise queries on specific fields, which reduces indexing costs and enhances query efficiency. For more information, see Create Index.

Environment Variables

1. Configure Simple Log Service when creating an application.

   Configure through the Container Console

   1. Log on to the Container Service Management Console, and in the left-side navigation pane, select Cluster.

   2. On the Cluster List page, click the name of the target cluster, and then in the left-side navigation pane, select Workloads > Stateless.

   3. On the Stateless page, set the namespace in the Namespace drop-down list at the top, and then click Create With Image in the upper-right corner.

   4. On the Application Basic Information tab, set the Application Name, click Next, and go to the Container Configuration page to set the image name.

      This section describes only the configurations related to Simple Log Service. For more information about other application configurations, see Create a Stateless Deployment.

   5. In the Log Configuration area, configure log-related information.

      1. Set the collection configuration.

         Click Collection Configuration to create a new collection configuration. Each collection configuration consists of two items: Logstore and Log Path In Container.

         • Logstore: Specify the name of the Logstore that is used to store the collected log data. If the Logstore does not exist, ACK automatically creates a Logstore in the Simple Log Service project that is associated with your ACK cluster.

           Note

           The default log retention period of Logstores is 90 days.

         • Log Path in Container: Specify the path from which you want to collect log data. For example, use /usr/local/tomcat/logs/catalina.*.log to collect the text logs of Tomcat.

           All settings are added as configuration entries to the corresponding Logstore. By default, logs are collected in simple mode (by row).

      2. Set a custom tag.

         Click Custom Tag to create a new custom tag. Each custom tag is a key-value pair that is appended to the collected logs. You can use it to tag the log data of the container, such as the version number.

   6. After you complete all configurations, click Next in the upper-right corner to proceed to the next step.

      For more information about the subsequent steps, see Create a Stateless Deployment.

   Configure through a YAML Template

   1. Log on to the Container Service Management Console, and in the left-side navigation pane, select Cluster List.

   2. On the Cluster List page, click the name of the target cluster, and then in the left-side navigation pane, select Workloads > Stateless.

   3. On the Stateless page, set the namespace in the Namespace drop-down list at the top, and then click Create Resource With YAML in the upper-right corner.

   4. Configure the YAML file.

      The syntax of the YAML template is the same as the Kubernetes syntax. However, to specify collection configurations for containers, you need to use env to add Collection Configuration and Custom Tag for the container, and create corresponding volumeMounts and volumes based on the collection configuration. The following is a simple example of a pod:

      apiVersion: v1
      kind: Pod
      metadata:
        name: my-demo
      spec:
        containers:
        - name: my-demo-app
          image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
          env:
          # Configure environment variables
          - name: aliyun_logs_log-varlog
            value: /var/log/*.log
          - name: aliyun_logs_mytag1_tags
            value: tag1=v1
          # Configure volume mounting
          volumeMounts:
          - name: volumn-sls-mydemo
            mountPath: /var/log
          # If the pod is repetitively restarted, you can add a sleep command to the startup parameters of the pod
          command: ["sh", "-c"]  # Run commands in the shell
          args: ["sleep 3600"]   # Make the pod sleep 3,600 seconds (1 hour)
        volumes:
        - name: volumn-sls-mydemo
          emptyDir: {}

      1. Create your Collection Configuration and Custom Tag through environment variables. All environment variables related to configuration use ```html aliyun_logs_ as a prefix.

         • Create collection configurations in the following format:

           - name: aliyun_logs_log-varlog
             value: /var/log/*.log                        

           The example creates a collection configuration in the format of aliyun_logs_{key}, where the corresponding {key} is log-varlog.

           • aliyun_logs_log-varlog: This env indicates the creation of a Logstore named log-varlog, with a log collection path of /var/log/*.log. The corresponding log service collection configuration name is also log-varlog. The purpose is to collect the content of the /var/log/*.log file in the container into the log-varlog Logstore.

         • Create Custom Tag in the following format:

           - name: aliyun_logs_mytag1_tags
             value: tag1=v1                       

           After a tag is added, the tag is automatically appended to the log data that is collected from the container. The mytag1 is any name that does not contain an underscore (_).

      2. If your collection configuration specifies a collection path other than stdout, you need to create the corresponding volumeMounts in this section.

         The example collection configuration adds collection for /var/log/*.log, so the corresponding volumeMounts for /var/log is added.

   5. After you finish writing the YAML, click Create to submit the configuration to the Kubernetes cluster for execution.

2. Configure advanced parameters for environment variables.

   You can configure container environment variables to customize log collection. You can set advanced parameters based on your actual needs to meet specific log collection requirements.

   Important

   You cannot use environment variables to configure log collection in edge computing scenarios.

   Field	Description	Example	Notes
   aliyun_logs_{key}	Required. {key} can contain only lowercase letters, digits, and hyphens (-). If the specified aliyun_logs_{key}_logstore does not exist, a Logstore named {key} is created to store the collected log data. If the value is stdout, it indicates that the stdout of a container is collected. Other values indicate the log path inside the container.	- name: aliyun_logs_catalina value: stdout - name: aliyun_logs_access-log value: /var/log/nginx/access.log	The default log collection mode is simple mode. If you want to parse log content, it is recommended to use the Simple Log Service console or configure it through CRD. {key} specifies the name of the Logtail configuration. The configuration name must be unique in the Kubernetes cluster.
   aliyun_logs_{key}_tags	Optional. The value must be in the following format: {tag-key}={tag-value}. It is used to add tags to log data.	- name: aliyun_logs_catalina_tags value: app=catalina	N/A.
   aliyun_logs_{key}_project	Optional. This variable specifies a project in Simple Log Service. If this environment variable is not configured, the project that you specified when you created the cluster is used.	- name: aliyun_logs_catalina_project value: my-k8s-project	The project must be deployed in the same region as Logtail.
   aliyun_logs_{key}_logstore	Optional. This variable specifies a Logstore in Simple Log Service. If this environment variable is not configured, the Logstore is named {key}.	- name: aliyun_logs_catalina_logstore value: my-logstore	N/A.
   aliyun_logs_{key}_shard	Optional. The variable specifies the number of shards of the Logstore. Valid values: 1 to 10. If this environment variable is not configured, the default value is 2. Note If the Logstore that you specify already exists, this variable does not take effect.	- name: aliyun_logs_catalina_shard value: '4'	N/A.
   aliyun_logs_{key}_ttl	Optional. The variable specifies the log retention period. Valid values: 1 to 3650. To retain log data permanently, set the value to 3650. If this environment variable is not configured, the default retention period is 90 days. Note If the Logstore that you specify already exists, this variable does not take effect.	- name: aliyun_logs_catalina_ttl value: '3650'	N/A.
   aliyun_logs_{key}_machinegroup	Optional. This variable specifies the node group in which the application is deployed. If this environment variable is not configured, the default node group is the one in which Logtail is deployed. For more information about this parameter, see Customization Requirement 2: Collect Data from Different Applications into Different Projects.	- name: aliyun_logs_catalina_machinegroup value: my-machine-group	N/A.
   aliyun_logs_{key}_logstoremode	Optional. This variable specifies the type of the Logstore in Simple Log Service. If this parameter is not specified, the default value is standard. Valid values: Note If the Logstore that you specify already exists, this variable does not take effect. standard: Supports the one-stop data analysis feature of Simple Log Service. It is suitable for scenarios such as real-time monitoring, interactive analysis, and building a complete observability system. query: Supports high-performance queries. The index traffic cost is about half of that of standard, but SQL analysis is not supported. It is suitable for scenarios with large data volumes, long storage periods (weeks or months), and no log analysis.	- name: aliyun_logs_catalina_logstoremode value: standard - name: aliyun_logs_catalina_logstoremode value: query	This parameter requires the logtail-ds image version to be 1.3.1 or later.

   • Customization Requirement 1: Collect Data From Multiple Applications Into The Same Logstore

     If you need to collect data from multiple applications into the same Logstore, you can set the aliyun_logs_{key}_logstore parameter. For example, the following configuration collects the stdout of two applications into stdout-logstore.

     In the example, the {key} of Application 1 is app1-stdout, and the {key} of Application 2 is app2-stdout.

     The environment variables configured for Application 1 are as follows:

     # Configure environment variables
         - name: aliyun_logs_app1-stdout
           value: stdout
         - name: aliyun_logs_app1-stdout_logstore
           value: stdout-logstore

     The environment variables configured for Application 2 are as follows:

     # Configure environment variables
         - name: aliyun_logs_app2-stdout
           value: stdout
         - name: aliyun_logs_app2-stdout_logstore
           value: stdout-logstore

   • Customization Requirement 2: Collect Data From Different Applications Into Different Projects

     If you need to collect data from different applications into multiple projects, perform the following steps:

     1. Create a machine group in each project, select a custom ID, and name it k8s-group-{cluster-id}, where {cluster-id} is your cluster ID. The machine group name can be customized.

     2. Configure the project, logstore, and machinegroup information in the environment variables of each application. The machine group name is the name of the machine group you created in the previous step.

        In the following example, the {key} of Application 1 is app1-stdout, and the {key} of Application 2 is app2-stdout. If the two applications are deployed in the same ACK cluster, you can use the same machine group for the applications.

        The environment variables configured for Application 1 are as follows:

        # Configure environment variables
            - name: aliyun_logs_app1-stdout
              value: stdout
            - name: aliyun_logs_app1-stdout_project
              value: app1-project
            - name: aliyun_logs_app1-stdout_logstore
              value: app1-logstore
            - name: aliyun_logs_app1-stdout_machinegroup
              value: app1-machine-group

        The environment variables configured for Application 2 are as follows:

        # Configure environment variables for Application 2
            - name: aliyun_logs_app2-stdout
              value: stdout
            - name: aliyun_logs_app2-stdout_project
              value: app2-project
            - name: aliyun_logs_app2-stdout_logstore
              value: app2-logstore
            - name: aliyun_logs_app2-stdout_machinegroup
              value: app1-machine-group