Collect text logs of the ACK cluster (deploy LoongCollector in the DaemonSet mode)

Create the AliyunPipelineConfig custom resource to automatically create a LoongCollector collection configuration.

1. Log on to the ACK console.

2. In the left-side navigation pane, click Cluster List.

3. On the Cluster List page, click More in the Actions column of the target cluster, then select Manage Cluster.

4. Create a file named example-k8s-file.yaml.

   The sample YAML file provided below is designed to collect multi-line text from the test.LOG file located at /data/logs/app_1 on a Pod labeled with app: ^(.*test.*)$ within the default namespace. It then forwards this data to a Logstore called k8s-file, which is automatically created within the k8s-log-test project. You will need to adjust the following parameters in the YAML to match your specific requirements:

   1. project, example: k8s-log-test.

      Log on to the Simple Log Service console, and determine the project name generated by LoongCollector. The format is typically k8s-log-<YOUR_CLUSTER_ID>.

   2. IncludeK8sLabel, example: app: ^(.*test.*)$. This filters the target Pod labels. The current condition specifies Pods with label key 'app' and values containing 'test' will be collected.

      Note

      To collect all Pods with names containing 'test' in the cluster, replace IncludeK8sLabel with K8sContainerRegex and configure the value with a wildcard, such as: K8sContainerRegex: ^(.test.)$.

   3. FilePaths, example: /data/logs/app_1/**/test.LOG. For more details, see Container file path mapping.

   4. Endpoint and Region, example: cn-hangzhou.log.aliyuncs.com and cn-hangzhou.

   For details about the config item in the YAML file, including supported input, output, processing plug-in types, and container filtering methods, see PipelineConfig. For complete YAML parameter details, see CR parameter description.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   kind: ClusterAliyunPipelineConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster. This name is also the name of the created collection configuration. If the name is duplicated, it will not take effect.
     name: example-k8s-file
   spec:
     # Specify the target project
     project:
       name: k8s-log-test
     logstores:
       # Create a Logstore named k8s-file
       - name: k8s-file
     # Define the collection configuration
     config:
       # Sample log (optional)
       sample: |
         2024-06-19 16:35:00 INFO test log
         line-1
         line-2
         end
       # Define input plug-ins
       inputs:
         # Use the input_file plug-in to collect multi-line text logs from containers
         - Type: input_file
           # File path in the container
           FilePaths:
             - /data/logs/app_1/**/test.LOG
           # Enable container discovery feature.
           EnableContainerDiscovery: true
           # Add conditions to filter container information. Multiple options are evaluated by using a logical AND.
           CollectingContainersMeta: true
           ContainerFilters:
             # Specify the namespace to which the container to be collected belongs. Regular expression matching is supported.
             K8sNamespaceRegex: default
             # Specify the name of the container to be collected. Regular expression matching is supported.
             IncludeK8sLabel:
               app: ^(.*app.*)$
           # Enable multi-line log collection. Delete this configuration for single-line log collection
           Multiline:
             # Specify the custom mode to match the beginning of the first line of a log based on a regular expression
             Mode: custom
             # Configure the regular expression for the beginning of a line
             StartPattern: '\d+-\d+-\d+\s\d+:\d+:\d+'
       # Define processing plug-ins
       processors:
         # Use the processor_parse_regex_native plug-in to parse logs
         - Type: processor_parse_regex_native
           # Name of the source field
           SourceKey: content
           # Regular expression for parsing. Use capturing groups to extract fields
           Regex: (\d+-\d+-\d+\s\S+)(.*)
           # List of extracted fields
           Keys: ["time", "detail"]
       # Define output plug-ins
       flushers:
         # Use the flusher_sls plug-in to send logs to a specific Logstore.
         - Type: flusher_sls
           # Make sure that the Logstore exists
           Logstore: k8s-file
           # Make sure that the endpoint is valid
           Endpoint: cn-beijing.log.aliyuncs.com
           Region: cn-beijing
           TelemetryType: logs

5. Execute kubectl apply -f example-k8s-file.yaml. LoongCollector begins collecting text logs from the Pod and sends them to Simple Log Service.

Create the AliyunLogConfig custom resource to automatically create a LoongCollector collection configuration.

1. Log on to the ACK console.

2. In the left-side navigation pane, click Cluster List.

3. On the Cluster List page, click More in the Actions column of the target cluster, then select Manage Cluster.

4. Create a file named example-k8s-file.yaml.

   This YAML script establishes a LoongCollector collection configuration called example-k8s-file. It retrieves the contents of the test.LOG file located at /data/logs/app_1 from all containers with names beginning with app within the cluster. The data is collected in simple text format and transmitted to a Logstore named k8s-file, which is automatically created within a project called k8s-log-<YOUR_CLUSTER_ID>.

   You need to adjust the file path within the container to reflect your actual circumstances. For more information, see Container file path mapping.

   • logPath: Specifies the path for log collection. For example, /data/logs/app_1.

   • filePattern: Designates the name of the log file to be collected. For example, test.LOG.

   For more information on the logtailConfig item provided by LoongCollector in the YAML file, including supported inputs, outputs, processing plug-in types, and container filtering methods, see AliyunLogConfigDetail. For a comprehensive list of YAML parameters, refer to CR parameter description.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster.
     name: example-k8s-file
     # Specify the namespace to which the resource belongs.
     namespace: kube-system
   spec:
     # Specify the name of the target project. If you leave this parameter empty, the project named k8s-log-<your_cluster_id> is used.
     # project: k8s-log-test
     # Specify the name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore.
     logstore: k8s-file
     # Specify the collection configuration.
     logtailConfig:
       # Specify the type of the data source. If you want to collect text logs, set the value to file.
       inputType: file
       # Specify the name of the collection configuration. The name must be the same as metadata.name.
       configName: example-k8s-file
       inputDetail:
         # Specify the simple mode to collect text logs.
         logType: common_reg_log
         # Specify the path to the log file.
         logPath: /data/logs/app_1
         # Specify the log file name. You can use wildcard characters such as asterisks (*) and question marks (?). Example: log_*.log.
         filePattern: test.LOG
         # If you want to collect container text logs, set the dockerFile parameter to true.
         dockerFile: true
         # Enable multi-line log collection. Delete this configuration for single-line log collection
         # Regular expression for the beginning of a line. This regular expression indicates the beginning of a log line.
         logBeginRegex: \d+-\d+-\d+.*
         # Specify the conditions that are used to filter containers.
         advanced:
           k8s:
             K8sPodRegex: '^(app.*)$'

5. Execute kubectl apply -f example-k8s-file.yaml. LoongCollector begins collecting text logs from the Pod and sends them to Simple Log Service.

1. Log on to the Simple Log Service console.

2. Select the project you used when installing the LoongCollector component from the Project list, such as k8s-log-<your_cluster_id>. On the Project page, click Logtail Configuration of the target Logstore, add a collection configuration, and click Kubernetes - File Access Now.

3. On the Machine Group Configuration page, select the k8s-group-${your_k8s_cluster_id} machine group in the ACK Daemonset mode of the K8s scenario and click > to add it to the Application Machine Group, and then click Next.

4. Create a collection configuration. Fill in the required configurations as described below and click Next. It takes about 1 minute for the collection configuration to take effect. Please be patient. Only the main configurations are introduced here. For detailed configurations, see Logtail collection configuration.

   • Global Configuration

     Enter the configuration name in Global Configuration.

     

   • Input Configuration

     • Logtail Deployment Mode: Select DaemonSet.

     • File Path Type: Choose whether the file path to be collected is a path in the container or on the host. For containers that mount volumes using the hostPath method, if you want to directly collect log files mapped on the host, select Host Path. Otherwise, select Path in Container.

     • File Path: The path for log collection. The log path must start with a forward slash (/). For example, /data/wwwlogs/main/**/*.Log in the following figure indicates files with the suffix .Log in the /data/wwwlogs/main directory. If you need to set the maximum depth of the monitored log directory, that is, the maximum directory depth matched by the wildcard ** in File Path, you can modify the value of the maximum directory monitoring depth. 0 indicates that only the current directory is monitored.

5. Create Index and Preview Data: Simple Log Service enables full-text indexing by default. At this time, queries will index all fields in the logs. You can also manually create field indexes based on the collected logs, or click Auto-generate Index. Simple Log Service will generate field indexes, allowing you to perform precise queries on specific fields, thereby reducing index costs and improving query efficiency. For more information, see Create index.

1. Configure Simple Log Service when creating an application.

   Configure through the Container Console

   Configure through a YAML Template

   1. Log on to the Container Service Management Console, and in the left-side navigation pane, select Cluster.

   2. On the Cluster List page, click the name of the target cluster, and then in the left-side navigation pane, select Workloads > Stateless.

   3. On the Stateless page, set the namespace in the Namespace drop-down list at the top, and then click Create With Image in the upper-right corner.

   4. On the Application Basic Information tab, set the Application Name, click Next, and go to the Container Configuration page to set the image name.

      This section describes only the configurations related to Simple Log Service. For more information about other application configurations, see Create a Stateless Deployment.

   5. In the Log Configuration area, configure log-related information.

      1. Set the collection configuration.

         Click Collection Configuration to create a new collection configuration. Each collection configuration consists of two items: Logstore and Log Path In Container.

         • Logstore: Specify the name of the Logstore that is used to store the collected log data. If the Logstore does not exist, ACK automatically creates a Logstore in the Simple Log Service project that is associated with your ACK cluster.

           Note

           The default log retention period of Logstores is 90 days.

         • Log Path in Container: Specify the path from which you want to collect log data. For example, use /usr/local/tomcat/logs/catalina.*.log to collect the text logs of Tomcat.

           All settings are added as configuration entries to the corresponding Logstore. By default, logs are collected in simple mode (by row).

      2. Set a custom tag.

         Click Custom Tag to create a new custom tag. Each custom tag is a key-value pair that is appended to the collected logs. You can use it to tag the log data of the container, such as the version number.

   6. After you complete all configurations, click Next in the upper-right corner to proceed to the next step.

      For more information about the subsequent steps, see Create a Stateless Deployment.

   1. Log on to the Container Service Management Console, and in the left-side navigation pane, select Cluster List.

   2. On the Cluster List page, click the name of the target cluster, and then in the left-side navigation pane, select Workloads > Stateless.

   3. On the Stateless page, set the namespace in the Namespace drop-down list at the top, and then click Create Resource With YAML in the upper-right corner.

   4. Configure the YAML file.

      The syntax of the YAML template is the same as the Kubernetes syntax. However, to specify collection configurations for containers, you need to use env to add Collection Configuration and Custom Tag for the container, and create corresponding volumeMounts and volumes based on the collection configuration. The following is a simple example of a pod:

      apiVersion: v1
      kind: Pod
      metadata:
        name: my-demo
      spec:
        containers:
        - name: my-demo-app
          image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
          env:
          # Configure environment variables
          - name: aliyun_logs_log-varlog
            value: /var/log/*.log
          - name: aliyun_logs_mytag1_tags
            value: tag1=v1
          # Configure volume mounting
          volumeMounts:
          - name: volumn-sls-mydemo
            mountPath: /var/log
          # If the pod is repetitively restarted, you can add a sleep command to the startup parameters of the pod
          command: ["sh", "-c"]  # Run commands in the shell
          args: ["sleep 3600"]   # Make the pod sleep 3,600 seconds (1 hour)
        volumes:
        - name: volumn-sls-mydemo
          emptyDir: {}

      1. Create your Collection Configuration and Custom Tag through environment variables. All environment variables related to configuration use ```html aliyun_logs_ as a prefix.

         • Create collection configurations in the following format:

           - name: aliyun_logs_log-varlog
             value: /var/log/*.log                        

           The example creates a collection configuration in the format of aliyun_logs_{key}, where the corresponding {key} is log-varlog.

           • aliyun_logs_log-varlog: This env indicates the creation of a Logstore named log-varlog, with a log collection path of /var/log/*.log. The corresponding log service collection configuration name is also log-varlog. The purpose is to collect the content of the /var/log/*.log file in the container into the log-varlog Logstore.

         • Create Custom Tag in the following format:

           - name: aliyun_logs_mytag1_tags
             value: tag1=v1                       

           After a tag is added, the tag is automatically appended to the log data that is collected from the container. The mytag1 is any name that does not contain an underscore (_).

      2. If your collection configuration specifies a collection path other than stdout, you need to create the corresponding volumeMounts in this section.

         The example collection configuration adds collection for /var/log/*.log, so the corresponding volumeMounts for /var/log is added.

   5. After you finish writing the YAML, click Create to submit the configuration to the Kubernetes cluster for execution.

2. Configure advanced parameters of environment variables.

   Environment variables support multiple configuration parameters for collection. You can set advanced parameters based on your actual needs to meet special requirements for log collection.

   Important

   Configuring log collection via environment variables is not suitable for edge computing scenarios.

   Field	Description	Example	Notes
   aliyun_logs_{key}	Required. {key} can contain only lowercase letters, digits, and hyphens (-). If the specified aliyun_logs_{key}_logstore does not exist, a Logstore named {key} is created. If the value is stdout, the standard output of the container is collected. Other values indicate the log path in the container.	- name: aliyun_logs_catalina value: stdout - name: aliyun_logs_access-log value: /var/log/nginx/access.log	The default log collection mode is simple mode. If you want to parse log content, it is recommended to use the Simple Log Service console or configure it through CRD. {key} specifies the name of the Logtail configuration in Simple Log Service. The configuration name must be unique in the Kubernetes cluster.
   aliyun_logs_{key}_tags	Optional. The value must be in the following format: {tag-key}={tag-value}. It is used to add tags to log data.	- name: aliyun_logs_catalina_tags value: app=catalina	None
   aliyun_logs_{key}_project	Optional. The variable specifies a project in Simple Log Service. If this environment variable is not configured, the project that you specified when you installed the cluster is used.	- name: aliyun_logs_catalina_project value: my-k8s-project	The project must be deployed in the same region as Logtail.
   aliyun_logs_{key}_logstore	Optional. The variable specifies a Logstore in Simple Log Service. If this environment variable is not configured, the Logstore is named {key}.	- name: aliyun_logs_catalina_logstore value: my-logstore	None
   aliyun_logs_{key}_shard	Optional. The variable specifies the number of shards of the Logstore. Valid values: 1 to 10. If this environment variable is not configured, the default value is 2. Note If the Logstore that you specify already exists, this variable does not take effect.	- name: aliyun_logs_catalina_shard value: '4'	None
   aliyun_logs_{key}_ttl	Optional. The variable specifies the log retention period. Valid values: 1 to 3650. To retain log data permanently, set the value to 3650. If this environment variable is not configured, the default retention period is 90 days. Note If the Logstore that you specify already exists, this variable does not take effect.	- name: aliyun_logs_catalina_ttl value: '3650'	None
   aliyun_logs_{key}_machinegroup	Optional. The variable specifies the node group in which the application is deployed. If this environment variable is not configured, the default node group is the one in which Logtail is deployed. For more information about this parameter, see Collect text logs of Alibaba Cloud ACK clusters (deploy Logtail in DaemonSet mode).	- name: aliyun_logs_catalina_machinegroup value: my-machine-group	None
   aliyun_logs_{key}_logstoremode	Optional. The variable specifies the type of the Logstore in Simple Log Service. If this parameter is not specified, the default value is standard. Valid values: Note If the Logstore that you specify already exists, this variable does not take effect. standard: Supports one-stop data analysis features of Simple Log Service. Applicable to scenarios such as real-time monitoring, interactive analysis, and building a complete observability system. query: Supports high-performance queries. The index traffic cost is about half of that of standard, but SQL analysis is not supported. Applicable to scenarios with large data volumes, long storage periods (weeks or months), and no log analysis.	- name: aliyun_logs_catalina_logstoremode value: standard - name: aliyun_logs_catalina_logstoremode value: query	None

   • Customization Requirement 1: Collect Data From Multiple Applications Into The Same Logstore

     If you need to collect data from multiple applications into the same Logstore, you can set the aliyun_logs_{key}_logstore parameter. For example, the following configuration collects the stdout of two applications into stdout-logstore.

     In the example, the {key} of Application 1 is app1-stdout, and the {key} of Application 2 is app2-stdout.

     The environment variables configured for Application 1 are as follows:

     # Configure environment variables
         - name: aliyun_logs_app1-stdout
           value: stdout
         - name: aliyun_logs_app1-stdout_logstore
           value: stdout-logstore

     The environment variables configured for Application 2 are as follows:

     # Configure environment variables
         - name: aliyun_logs_app2-stdout
           value: stdout
         - name: aliyun_logs_app2-stdout_logstore
           value: stdout-logstore

   • Customization Requirement 2: Collect Data From Different Applications Into Different Projects

     If you need to collect data from different applications into multiple projects, perform the following steps:

     1. Create a machine group in each project, select a custom ID, and name it k8s-group-{cluster-id}, where {cluster-id} is your cluster ID. The machine group name can be customized.

     2. Configure the project, logstore, and machinegroup information in the environment variables of each application. The machine group name is the name of the machine group you created in the previous step.

        In the following example, the {key} of Application 1 is app1-stdout, and the {key} of Application 2 is app2-stdout. If the two applications are deployed in the same ACK cluster, you can use the same machine group for the applications.

        The environment variables configured for Application 1 are as follows:

        # Configure environment variables
            - name: aliyun_logs_app1-stdout
              value: stdout
            - name: aliyun_logs_app1-stdout_project
              value: app1-project
            - name: aliyun_logs_app1-stdout_logstore
              value: app1-logstore
            - name: aliyun_logs_app1-stdout_machinegroup
              value: app1-machine-group

        The environment variables configured for Application 2 are as follows:

        # Configure environment variables for Application 2
            - name: aliyun_logs_app2-stdout
              value: stdout
            - name: aliyun_logs_app2-stdout_project
              value: app2-project
            - name: aliyun_logs_app2-stdout_logstore
              value: app2-logstore
            - name: aliyun_logs_app2-stdout_machinegroup
              value: app1-machine-group