Install Logtail components in an ACK cluster - Simple Log Service

This topic describes how to install and upgrade Logtail components in a Container Service for Kubernetes (ACK) cluster.

Background information

To collect container logs from a Kubernetes cluster, you must install Logtail components.

When you install Logtail components, Simple Log Service automatically completes the following operations:

Create a ConfigMap named alibaba-log-configuration. The ConfigMap contains the configuration information of Simple Log Service, such as projects.
Optional. Create a custom resource definition (CRD) named AliyunLogConfig.
Optional. Create a Deployment named alibaba-log-controller. The Deployment is used to monitor the changes in the AliyunLogConfig CRD and the creation of Logtail configurations.
Create a DaemonSet named logtail-ds to collect logs from nodes.

Install Logtail

Important

If you are using an ACK dedicated cluster or an ACK managed cluster, you can follow the instructions in this section to install Logtail components in your ACK cluster.
For more information about how to collect text logs, standard output (stdout), and standard errors (stderr) from containers in a serverless Kubernetes (ASK) cluster, see Use pod environment variables to collect application logs.
If your ACK cluster and Simple Log Service resources belong to different Alibaba Cloud accounts, refer to Install Logtail components in a self-managed Kubernetes cluster.

Install Logtail components in an existing ACK cluster

Log on to the ACK console.
On the Clusters page, find the cluster in which you want to install Logtail components and choose More > Operations > Manage Components in the Actions column.
On the Logs and Monitoring tab, find logtail-ds and click Install.
After logtail-ds is installed, Simple Log Service automatically creates a project named k8s-log-${your_k8s_cluster_id}, a machine group named k8s-group-${your_k8s_cluster_id}, and a logstore named config-operation-log in the project.
Important
Do not delete the config-operation-log logstore.

Install Logtail components when you create an ACK cluster

Log on to the ACK console.
On the Clusters page, click Create Kubernetes Cluster.
In the Component Configurations step, select Enable Log Service.
Note
In this example, only the steps that are required to enable Simple Log Service are provided. For more information about how to create an ACK cluster, see Create an ACK managed cluster.
If you select Enable Log Service, the system prompts you to create a Simple Log Service project. For more information about how logs are managed in Simple Log Service, see Project. You can use one of the following methods to create a project:
- Select Project
  You can select an existing project to manage the container logs that are collected.
- Create Project
  Simple Log Service automatically creates a project named k8s-log-{ClusterID} to manage the container logs that are collected. ClusterID specifies the unique ID of the ACK cluster that you create.

After the Logtail components are installed, a machine group named k8s-group-${your_k8s_cluster_id} and a logstore named config-operation-log are automatically created in your project.

Important

Do not delete the config-operation-log logstore.

View the status, version number, and IP address of Logtail

View the status of Logtail

Run the following command to view the status of Logtail:

kubectl get po -n kube-system | grep logtail

The following output is returned:

NAME            READY     STATUS    RESTARTS   AGE
logtail-ds-gb92k   1/1       Running   0          2h
logtail-ds-wm7lw   1/1       Running   0          4d

View the version number and IP address of Logtail

Run the following command to view the version number and IP address of Logtail:

kubectl exec logtail-ds-gb92k -n kube-system cat /usr/local/ilogtail/app_info.json

The following output is returned:

{
   "UUID" : "",
   "hostname" : "logtail-ds-gb92k",
   "instance_id" : "0EBB2B0E-0A3B-11E8-B0CE-0A58AC140402_172.20.4.2_1517810940",
   "ip" : "192.0.2.0",
   "logtail_version" : "0.16.2",
   "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
   "update_time" : "2021-02-05 06:09:01"
}

Upgrade Logtail

Back up files

Important

An upgrade requires a few seconds to complete. During an upgrade, the Logtail container is restarted, which may cause a small amount of data to be collected again or lost during collection.

Before you upgrade the Logtail components, we recommend that you back up the description files that are related to the Logtail components.

kubectl get ds -n kube-system logtail-ds -o yaml > logtail-ds.yaml
kubectl get deployment -n kube-system alibaba-log-controller -o yaml > alibaba-log-controller.yaml
kubectl get crd aliyunlogconfigs.log.alibabacloud.com -o yaml > aliyunlogconfigs-crd.yaml
kubectl get cm -n kube-system alibaba-log-configuration -o yaml > alibaba-log-configuration.yaml
kubectl get aliyunlogconfigs --all-namespaces -o yaml > aliyunlogconfigs-cr.yaml

Upgrade components

We recommend that you use the automatic upgrade method in common scenarios. If you have modified parameters such as environment variables in the logtail-ds DaemonSet or alibaba-log-controller Deployment, we recommend that you use the manual upgrade method to retain your modifications.

Automatic upgrade

Important

If you use the automatic upgrade method, your modifications to the parameters in the logtail-ds DaemonSet and alibaba-log-controller Deployment are not retained.

Log on to the ACK console.
On the Clusters page, find the cluster in which you want to install Logtail components and choose More > Operations > Manage Components in the Actions column.
On the Logs and Monitoring tab, find logtail-ds and click Upgrade.
In the Update dialog box, click OK.
Important
If the component cannot be upgraded to the most up-to-date Logtail version, the Kubernetes version of your cluster is outdated. In this case, you must upgrade the Kubernetes version of your cluster first or use the manual upgrade method.
After the upgrade is performed, you can view the status of each logtail-ds pod in the ACK console. If each logtail-ds pod is in the running state, the upgrade is successful.

Manual upgrade

Important

If you use the manual upgrade method, your existing configurations are not updated, and some features may not be supported.

A manual upgrade covers both logtail-ds and alibaba-log-controller. In most cases, you need to only upgrade logtail-ds to obtain the collection capabilities provided in the most up-to-date version of Logtail. If you want to obtain the CRD-based collection capabilities provided in the most up-to-date version of Logtail, you must also upgrade alibaba-log-controller. The following procedure shows how to upgrade logtail-ds.

Log on to the ACK console.
On the Clusters page, find the cluster in which you want to install Logtail components and choose More > Operations > Manage Components in the Actions column.
Choose Workloads > DaemonSets.
Note
If you want to upgrade alibaba-log-controller, choose Workloads > Deployments. Then, set Namespace to kube-system and find alibaba-log-controller.
Set Namespace to kube-system. Then, find logtail-ds and click Edit in the Actions column.
Check whether the required environment variables exist.
If the ALIYUN_LOGTAIL_CONFIG, ALIYUN_LOGTAIL_USER_ID, or ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable does not exist, your Logtail version is outdated. You can submit a ticket for technical support.
Click Select Image Version to the right of Image Version.
In the Image Version dialog box, click the most up-to-date version and click OK.
In the right-side pane of the page that appears, click Update.
After the upgrade is performed, you can view the status of each logtail-ds pod in the ACK console. If each logtail-ds pod is in the running state, the upgrade is successful.

Upgrade the latest version of Logtail

The YAML file used in the Logtail version named latest is outdated. If you use this version, issues may occur during an upgrade or when new features are used. We recommend that you upgrade the latest version to the most up-to-date version. Perform the following steps:

Store the existing AliyunLogConfig CRD.
Replace log-crds.yaml based on your business scenario.
```
kubectl get AliyunLogConfig -A -o yaml > log-crds.yaml
```
Uninstall logtail-ds.
On the Logs and Monitoring tab in the ACK console, find logtail-ds and click Uninstall. For more information, see Uninstall Logtail.
Install logtail-ds.
On the Logs and Monitoring tab in the ACK console, find logtail-ds and click Install. For more information, see Install Logtail.
Deploy the stored AliyunLogConfig CRD.
Replace log-crds.yaml based on your business scenario.
```
kubectl apply -f  log-crds.yaml
```

Roll back an upgrade

The following procedure shows how to roll back to a specific version.

Note

The YAML files that are backed up before an upgrade contain redundant information. Before you can restore the configurations of Logtail, you must manually delete the redundant information. You can use the kubectl-neat tool to delete the redundant information. You must delete the following fields: metadata.creationTimestamp, metadata.generation, metadata.resourceVersion, metadata.uid, and status.

Check whether to retain the new configurations of Logtail after the upgrade.
If you do not need to retain the new configurations of Logtail after the upgrade, delete the configurations.

Delete redundant information from the backup files.

cat logtail-ds.yaml | kubectl-neat > neat-logtail-ds.yaml
cat alibaba-log-controller.yaml | kubectl-neat > neat-alibaba-log-controller.yaml
cat aliyunlogconfigs-crd.yaml | kubectl-neat > neat-aliyunlogconfigs-crd.yaml
cat alibaba-log-configuration.yaml | kubectl-neat > neat-alibaba-log-configuration.yaml
cat aliyunlogconfigs-cr.yaml | kubectl-neat > neat-aliyunlogconfigs-cr.yaml

Use the backup files after you delete redundant information to restore the configurations of Logtail.

kubectl apply -f neat-logtail-ds.yaml
kubectl apply -f neat-alibaba-log-controller.yaml
kubectl apply -f neat-aliyunlogconfigs-crd.yaml
kubectl apply -f neat-alibaba-log-configuration.yaml
kubectl apply -f neat-aliyunlogconfigs-cr.yaml

Uninstall Logtail

Log on to the ACK console.
On the Clusters page, find the cluster in which you want to install Logtail components and choose More > Operations > Manage Components in the Actions column.
On the Logs and Monitoring tab, find logtail-ds and click Uninstall.
Click OK as prompted.

What to do next

Text logs
- Collect text logs from Kubernetes containers in DaemonSet mode
- Collect text logs from Kubernetes containers in Sidecar mode
Stdout and Stderr: Collect stdout and stderr from Kubernetes containers in DaemonSet mode
Kubernetes events: Collect Kubernetes events

Create Logtail configurations to collect container logs.

DaemonSet mode
- For more information about how to collect container logs by using CRDs, see Use CRDs to collect container logs in DaemonSet mode.
- For more information about how to collect container stdout and stderr in the Simple Log Service console, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode.
- For more information about how to collect container text logs in the Simple Log Service console, see Collect text logs from Kubernetes containers in DaemonSet mode.
Sidecar mode
- For more information about how to collect container logs by using CRDs, see Use CRDs to collect container text logs in Sidecar mode.
- For more information about how to collect container logs in the Simple Log Service console, see Use the Simple Log Service console to collect container text logs in Sidecar mode.

FAQ

How do I view the version of a container image?

You can visit one of the following image repositories to view the version of a container image:

How do I collect container logs from multiple Kubernetes clusters to the same Simple Log Service project?

Note

You can collect container logs from multiple Kubernetes clusters to the same Simple Log Service project only if the Kubernetes clusters reside in the same region.

If you want to collect container logs from multiple ACK clusters to the same Simple Log Service project, you must select the same project when you create the ACK clusters.

How do I view the logs of Logtail?

The logs of Logtail are stored in the ilogtail.LOG and logtail_plugin.LOG files in the /usr/local/ilogtail/ directory of a Logtail container.

The stdout and stderr of the container do not apply to the sample scenario. Ignore the following stdout and stderr:

start umount useless mount points, /shm$|/merged$|/mqueue$
umount: /logtail_host/var/lib/docker/overlay2/3fd0043af174cb0273c3c7869500fbe2bdb95d13b1e110172ef57fe840c82155/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/d5b10aa19399992755de1f85d25009528daa749c1bf8c16edff44beab6e69718/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/5c3125daddacedec29df72ad0c52fac800cd56c6e880dc4e8a640b1e16c22dbe/merged: must be superuser to unmount
......
xargs: umount: exited with status 255; aborting
umount done
start logtail
ilogtail is running
logtail status:
ilogtail is running

How do I view the status of Simple Log Service components in Kubernetes clusters?

Run the following commands:

kubectl get deploy alibaba-log-controller -n kube-system
kubectl get ds logtail-ds -n kube-system

What do I do if alibaba-log-controller fails to start?

Check whether alibaba-log-controller is installed by using the following method:

Run the installation command on the master node of your Kubernetes cluster.
Specify the ID of your Kubernetes cluster in the installation command.

If alibaba-log-controller fails to be installed by using the preceding method, run the kubectl delete -f deploy command to delete the installation template that is generated. Then, rerun the installation command.

How do I view the status of the logtail-ds DaemonSet in a Kubernetes cluster?

Run the kubectl get ds -n kube-system command to view the status of the Logtail-ds DaemonSet.

Note

The default namespace to which a Logtail container belongs is kube-system.

How do I view the operational logs of Logtail?

The operational logs of Logtail are stored in the ilogtail.LOG file in the /usr/local/ilogtail/ directory. If the log file is rotated, the generated files are compressed and stored as ilogtail.LOG.x.gz. Run the following command to view the logs:

kubectl exec logtail-ds-gb92k -n kube-system tail /usr/local/ilogtail/ilogtail.LOG

The following output is returned:

[2018-02-05 06:09:02.168693] [INFO] [9] [build/release64/sls/ilogtail/LogtailPlugin.cpp:104] logtail plugin Resume:start
[2018-02-05 06:09:02.168807] [INFO] [9] [build/release64/sls/ilogtail/LogtailPlugin.cpp:106] logtail plugin Resume:success
[2018-02-05 06:09:02.168822] [INFO] [9] [build/release64/sls/ilogtail/EventDispatcher.cpp:369] start add existed check point events, size:0
[2018-02-05 06:09:02.168827] [INFO] [9] [build/release64/sls/ilogtail/EventDispatcher.cpp:511] add existed check point events, size:0 cache size:0 event size:0 success count:0

How do I restart Logtail for a pod?

Stop Logtail.
In the following command, logtail-ds-gb92k -n specifies the container, and kube-system specifies the namespace. Configure the parameters based on your business scenario.
```
kubectl exec logtail-ds-gb92k -n kube-system /etc/init.d/ilogtaild stop
```
If the following output is returned, Logtail is stopped:
```
kill process Name: ilogtail pid: 7
kill process Name: ilogtail pid: 9
stop success
```
Start Logtail.
In the following command, logtail-ds-gb92k -n specifies the container, and kube-system specifies the namespace. Configure the parameters based on your business scenario.
```
kubectl exec logtail-ds-gb92k -n kube-system /etc/init.d/ilogtaild start
```
If the following output is returned, Logtail is started:
```
ilogtail is running 
```

References

For more information about how to install Logtail in a self-managed Kubernetes cluster, see Install Logtail components in a self-managed Kubernetes cluster.
To collect the logs of control plane components from an ACK managed cluster, you can enable the log collection feature in the ACK console. For more information, see Collect the logs of control plane components in ACK Pro clusters.
For more information about how to collect container logs from an ASK cluster, see Logs of control plane components in ASK Pro clusters.
For more information about how to troubleshoot issues when you create a Logtail configuration, see What do I do if a Logtail machine group has no heartbeats? The issues include that no machine group exists and no heartbeat is detected in machine groups.
For more information about how to troubleshoot errors when you collect logs, see What do I do if an error occurs when I use Logtail to collect logs from containers?
For more information about how to filter containers when you configure Logtail, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode.