Applications, containers, and basic architecture components within the cluster continuously generate a vast array of log data, complicating the process of log collection and management. ACK clusters integrate seamlessly with Alibaba Cloud Simple Log Service (SLS) to offer a streamlined log management solution.
For more information about the features and basic terms of SLS, see What is Simple Log Service and Glossary.
Log collection components introduction
The Logtail component, provided by SLS, is a specialized agent for log data collection within ACK clusters. You can deploy this component via the console to gather various log file types, including text logs, Binlog, and HTTP data, in a non-intrusive way. It supports log data collection for standard containers and Kubernetes clusters. Logtail is recommended for its ease of use, seamless integration, resource efficiency, and cost-effectiveness.
For more information about the functional differences between Logtail and open-source components, see Log Collection Agent Comparison.
Log collection principles
The following details the log collection methods and workflow.
Collection flow
In ACK clusters, the Logtail component acts as a data collection bridge, securely gathering and transmitting various log data to SLS for storage and analysis. This approach streamlines the configuration and management of log collection, enhancing the flexibility and efficiency of log management. The basic workflow and architecture for using Logtail to collect logs are as follows.
The entire workflow includes:
-
Deploy Logtail. For specific operations, see Install Logtail Component (Alibaba Cloud Kubernetes Cluster).
Upon installing the Logtail component, the AliyunLogConfig CRD is automatically created, and the alibaba-log-controller is deployed to link the AliyunLogConfig within the ACK cluster with the collection configuration in Simple Log Service.
-
Logtail collects data based on the configuration: Logtail retrieves the established collection configuration and operates accordingly.
After deploying Logtail, you must create a collection configuration. It supports collection through console, CRD configuration, and environment variable configuration. For specific operations, see Collect Kubernetes Container Text Logs through DaemonSet, Collect Kubernetes Container Standard Output through DaemonSet (Old Version), Collect Kubernetes Container Text Logs through Sidecar.
-
Report logs to SLS: Logtail uploads the collected log data to SLS in real-time.
Collection methods
In Kubernetes clusters, log collection is categorized into Sidecar and DaemonSet methods, each with distinct workflows and suitable scenarios.
-
DaemonSet method: The Logtail component is installed in the kube-system namespace, and the DaemonSet is named
logtail-ds. Logtail on each node is tasked with collecting data from all pods on that node, including standard output and file logs. For more information, see Collect Kubernetes Container Text Logs through DaemonSet. -
Sidecar method: In addition to the main application container, a Logtail Sidecar container is run on the node (manual injection into the application pod is required). The Logtail container shares the log volume with the main container. The Sidecar container method is used when employing a zero load node. For more information, see Collect Kubernetes Container Text Logs through Sidecar.
Classification |
DaemonSet Method |
Sidecar Method |
Scenarios |
Suitable for clusters with straightforward log categorization and relatively simple operations.
|
Ideal for large, diverse clusters. When different log collection configurations are required for each pod, this method is appropriate. However, it results in high log volume per pod and significant resource usage. |
Resources |
Deploys one Logtail pod per node. |
Deploys one Logtail container per pod. |
Deployment and Maintenance |
Maintenance is limited to the DaemonSet. |
Each pod requiring log collection must deploy a Logtail container. |
Log Type |
Captures standard output and text logs. |
Focuses on text logs. |
Supported log types
ACK clusters support the collection of log data from various sources, ensuring secure transmission to SLS for storage and analysis.
Log type | Description | References |
Application (container) logs | Supports rapid collection of container logs in ACK clusters, including the standard output of containers and text files within containers. |
|
Control plane component logs | Supports collecting logs from API Server, kube-scheduler, kube-controller-manager, cloud-controller-manager, Controlplane-Events, ALB Ingress Controller, and other control plane components. The collected logs will be sent to the specified Logstore. This not only facilitates centralized management and analysis of the running status of control plane components but also helps quickly troubleshoot issues, monitor component health, and verify the effects of custom parameter configurations. Each log service Logstore corresponds to a Kubernetes control plane component. | Collect control plane component logs in ACK managed clusters |
Network component logs |
| |
System plug-in logs | Supports log collection for Dashboard, network plug-ins, storage plug-ins, and other plug-ins. |
Limits
Limit item | Limit description |
Kubernetes version |
|
|
For additional limits, see Logtail Limit Description.
Billing description
The cost of using SLS in ACK clusters consists of cluster management fees and SLS charges.
-
Cluster management cost: Billed by ACK. Charges apply only to ACK Pro Edition,ACK Basic Edition andACK Dedicated Edition are not subject to fees. For more details, seeBilling Overview.
-
SLS cost: Charges are unified on a pay-as-you-go basis. Logstore costs are billed by SLS.
For Logstore billing methods, see Billing Overview.
FAQ
How to copy logs from one project to another project?
To copy access logs from project-a's logstore-a to project-b's newly created logstore-b for later unified querying and analysis within project-b, please refer to Copy Logstore data.
What to do if the log collection status is abnormal?
If log collection status is abnormal when using Logtail for container logs, troubleshoot and check the running status by referring to Troubleshooting Container Log Collection Exceptions.
Common issues about log loss, log deletion, reducing log storage costs, etc.
For guidance on adjusting log retention days, disabling log collection, and other related topics, see Logstore Related Issues.
Logtail related issues
For details on Logtail log collection delays, historical log collection methods, the time required for Logtail configuration updates to take effect, and more, please see Logtail basic issues.
Contact us
Should you have any questions while using ACK, please feel free to contact us.