Log overview

Updated at: 2025-02-17 03:24

Applications, containers, and infrastructure components in a cluster continuously generate a large amount of scattered log data, which complicates log collection and management. Container Service for Kubernetes (ACK) clusters are integrated with Simple Log Service (SLS) to provide a solution that simplifies log management.

Note

For more information about the features and terms of SLS, see What is Simple Log Service? and Terms.

Introduction to the log collection component

The Logtail component is an agent provided by SLS to collect logs from ACK clusters. You can install this component in the ACK console to collect multiple types of logs in an efficient manner, including text logs, binlogs, and HTTP data. This component supports log data collection for standard containers, Kubernetes clusters, and other container clusters. If you want to improve ease of use, integration convenience, resource efficiency, and cost control, we recommend that you use Logtail to collect logs.

For more information about the differences between Logtail and open-source components, see What are the differences among log collection agents?.

Log collection

The following section describes the log collection method and process.

Collection process

In ACK clusters, Logtail is serves as a bridge to securely collect and transmit various data to SLS for storage and analysis. This method simplifies log collection management and improves flexibility and efficiency. The following figure shows the process and architecture of using Logtail to collect logs.

image

The following section describes the processes of using Logtail:

  1. Deploy a Logtail container. For more information, see Install Logtail components in an ACK cluster.

    When you install Logtail, the AliyunLogConfig CRD is created and the alibaba-log-controller component is installed by default to associate the AliyunLogConfig CRD with the collection configurations in SLS.

  2. Logtail collects data based on the configuration. Logtail obtains and uses the collection configuration.

    After you install Logtail, you must create a collection configuration. You can use the console, CRD configuration, and environments variables to collect logs. For more information, see Collect text logs from Kubernetes containers in DaemonSet mode, Collect stdout and stderr from Kubernetes containers in DaemonSet mode (old version), and Collect text logs from Kubernetes containers in Sidecar mode.

  3. Logtail uploads the collected log data to SLS in real time.

Collection method

In a Kubernetes cluster, you can use the Sidecar or DaemonSet mode to collect logs. Each method has different collection processes and use scenarios.

  • DaemonSet mode: Logtail is installed in the kube-system namespace. The name of the DaemonSet is logtail-ds. Logtail of each node collects data, including standard output and files, from all pods that run on the node. For more information, see Collect text logs from Kubernetes containers in DaemonSet mode.

  • Sidecar mode: In addition to the primary container, the node runs the Sidecar container of Logtail. You must manually create the Logtail container in the business pod. The Logtail container and primary container share the log volume. When you use a virtual node, use the Sidecar mode. For more information, see Collect text logs from Kubernetes containers in Sidecar mode.

Category

DaemonSet mode

Sidecar mode

Category

DaemonSet mode

Sidecar mode

Scenarios

A cluster has clear log classification and a single function.

  • A cluster belongs to a single tenant and has a single function or few services.

  • You want to use the same log collection configuration for containers on the same node.

  • You want to collect the standard output stream.

  • Low resource consumption

Large, hybrid clusters

If you require different log collection configurations for each pod, you can use this method to collect logs. However, a single pod in this mode generates a large amount of logs and consumes a large amount of resources.

Resource usage

Run a Logtail container on each node.

Run a Logtail container on each pod.

Deployment and O&M

You need to only maintain the DaemonSet.

A Logtail container must be deployed for each pod that needs to collect logs.

Log type

Standard output and text logs.

Text logs.

Supported log types

In an ACK cluster, you can collect log data from different sources and securely transmit the log data to SLS for storage and analysis.

Log type

Description

Reference

Log type

Description

Reference

Application (container) logs

Log data from containers of the ACK cluster can be quickly collected. The log data includes the standard output and text files.

Control plane component logs

Logs are collected from control plane components, such as API server, kube-scheduler, kube-controller-manager, cloud-controller-manager, Controlplane-Events, and ALB Ingress controller. Logs are sent to a specific Logstore. This component facilitates centralized management and analysis of the running status of control plane components. This helps you quickly troubleshoot issues, monitor the health status of components, and verify the effect of custom parameter configurations.

Each Logstore corresponds to a Kubernetes control plane component.

Collect logs of control plane components in ACK managed clusters

Network component logs

  • The access logs of Ingress components can be collected to help you monitor traffic dynamics at the access layer of your application.

  • You can collect CoreDNS logs to identify and resolve issues, such as slow DNS resolution and abnormal access requests.

System component logs

Log collection for components, such as Dashboard, network plug-ins, and storage plug-ins, is supported.

Collect log files of system components

Limits

Item

Description

Item

Description

Kubernetes version

  • When you collect logs in DaemonSet mode, Kubernetes 1.10.0 or later is required. The HostToContainer mount propagation must be supported.

  • When you use a custom resource definition (CRD) to collect logs, Kubernetes 1.16.0 or later is required, and the alibaba-log-controller component must be installed.

    The apiextensions.k8s.io/v1beta1 API provided by Kubernetes 1.7.0 and later also supports CRDs. However, the stability of the API in the Beta version varies based on the specified Kubernetes version.

logtail-ds

  • logtail-ds: At least 0.1 cores and 256 MB of memory must be reserved on each node.

  • alibaba-log-controller: At least 0.05 cores and 100 MB of memory must be reserved on each node.

For more information about the limits of Logtail, see Logtail limits.

Billing

The costs incurred of SLS in the ACK cluster are the sum of cluster management fees and the SLS fees.

  • Cluster management fees are charged by ACK. The cluster management fee is charged only for ACK Pro clusters. No cluster management fee is charged for ACK Basic clusters or ACK dedicated clusters. For more information, see Billing.

  • The SLS fee is charged on a pay-as-you-go basis. The fees for the Logstore are charged by SLS.

    For more information about the billing methods of Logstores, see Billing overview.

FAQ

How do I copy logs from a project to another project?

You can replicate the access logs from the logstore-a Logstore of the project-a project to the logstore-b Logstore of the project-b project to query and analyze the data in the project-b project in a centralized manner. For more information, see Replicate data from a Logstore.

What do I do if the log collection status is abnormal?

When you use Logtail to collect logs from standard Docker containers or Kubernetes containers, errors may occur. For more information about how to troubleshoot the errors and check the running status of Logtail, see What do I do if errors occur when I collect logs from containers?

FAQ about log loss, log deletion, and log cost reduction

For more information about how to change the log retention period and disable log collection, see FAQ about Logstores.

Logtail-related issues

For more information about the collection latency of Logtail, how to collect historical logs, and effective time of Logtail configurations, see FAQs about Logtail.

Contact us

If you have questions about ACK, contact us.

  • On this page (1, M)
  • Introduction to the log collection component
  • Log collection
  • Collection process
  • Collection method
  • Supported log types
  • Limits
  • Billing
  • FAQ
  • How do I copy logs from a project to another project?
  • What do I do if the log collection status is abnormal?
  • FAQ about log loss, log deletion, and log cost reduction
  • Logtail-related issues
  • Contact us
Feedback