How to deploy a Logtail container to collect stdout and stderr from Docker containers - Simple Log Service

You can collect logs for query and analysis after Docker is deployed on the server. Docker logs are categorized into two types: standard output (stdout and stderr) and file logs. File logs refer to the logs generated within the container that are written to a specified directory on the server, while standard output indicates the real-time output stream of the container. This topic explains how to use Logtail to collect standard output of the container to Logstore.

Overview

In this topic, an Elastic Compute Service (ECS) instance that runs a Linux operating system is used. After you install Docker on the ECS instance, you can use Logtail to collect stdout and stderr from an application container that is deployed on the ECS instance to a specified Logstore.

Prerequisites

A project and a Logstore are created. For more information, see Manage a project and Manage a Logstore.
An ECS instance is created, Docker is installed on the instance, and the instance is in the running state. For more information, see Install and use Docker on a Linux instance.
The container from which you want to collect logs continuously generates logs.
Important
Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.

1. Deploy a Logtail container

1.1 Pull a Logtail image

Log on to the ECS console, find the ECS instance, and then click Connect in the Actions column. After the ECS instance is connected, run the docker -v command to view the Docker version. If information similar to the following code is returned, Docker is installed on the ECS instance.
Pull a Logtail image. For more information about the supported regions, see Appendix: Region names for Logtail installation. Replace cn-hangzhou with an actual value. If the ECS instance resides in a virtual private cloud (VPC), you must replace registry with registry-vpc.
```
docker pull registry.cn-hangzhou.aliyuncs.com/log-service/logtail
```

1.2 Deploy and start a Logtail container

Run the following command and replace the ${your_region_name}, ${your_aliyun_user_id}, and ${your_machine_group_user_defined_id} parameters with actual values:

docker run -d \
    -v /:/logtail_host:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    --env ALIYUN_LOGTAIL_CONFIG=/etc/ilogtail/conf/${your_region_name}/ilogtail_config.json \
    --env ALIYUN_LOGTAIL_USER_ID=${your_aliyun_user_id} \
    --env ALIYUN_LOGTAIL_USER_DEFINED_ID=${your_machine_group_user_defined_id} \
    registry.${your_region_name}.aliyuncs.com/log-service/logtail

If the command is successful, a container ID is generated. For example, if you want to deploy a Logtail container on a server in the machine group whose custom identifier is docker-log-demo, use the following sample code to start the container.

The following table describes the parameters in the code.

Parameter	Description
`--privileged`	You must configure the `--privileged` parameter. Otherwise, the `container text file busy` error may occur when you delete other containers. For more information, see Bug 1468249, Bug 1441737, and issue 34538. For CentOS 7.4 or later (excluding CentOS 8.0), you can remove the `--privileged` parameter and run the following command to set the `fs.may_detach_mounts` parameter to 1: `echo fs.may_detach_mounts=1 \| sudo tee /etc/sysctl.d/may_detach_mounts.conf sudo sysctl -p /etc/sysctl.d/may_detach_mounts.conf sysctl fs.may_detach_mounts`
`${your_region_name}`	The ID of the region where the project resides and the network type used by the project. For more information about the supported regions, see Appendix: Region names for Logtail installation. For more information about network types, see Select a network type. If the project uses the Internet, specify the value in the `region-internet` format. For example, if the project resides in the China (Hangzhou) region, set the parameter to cn-hangzhou-internet. If the project uses an Alibaba Cloud internal network, specify the value in the `region` format. For example, if the project resides in the China (Hangzhou) region, set the parameter to cn-hangzhou.
`${your_aliyun_user_id}`	The ID of the Alibaba Cloud account for which Simple Log Service is activated. For more information, see Obtain the ID of the Alibaba Cloud account for which Simple Log Service is activated.
`${your_machine_group_user_defined_id}`	The custom identifier of your machine group. Make sure that the identifier is unique in the region where the project resides. Example: `docker-log-demo`.

Important

You can configure custom settings for the startup parameters of the Logtail container only if the following conditions are met:

The following environment variables are configured: ALIYUN_LOGTAIL_USER_DEFINED_ID, ALIYUN_LOGTAIL_USER_ID, and ALIYUN_LOGTAIL_CONFIG.
The /var/run directory of the host on which the Logtail container runs is mounted on the /var/run directory of the Logtail container.
The root directory of the host is mounted on the /logtail_host directory of the Logtail container.
If the The parameter is invalid : uuid=none error is displayed in the /usr/local/ilogtail/ilogtail.LOG log file, you must create a file named product_uuid on the host. Then, enter a valid UUID in the created file, such as 169E98C9-ABC0-4A92-B1D2-AA6239C0D261, and mount the file on the /sys/class/dmi/id/product_uuid directory of the Logtail container.

2. Create a Logtail configuration

Log on to the Simple Log Service console. On the right side of the page that appears, click the Quick Data Import card. In the dialog box that appears, click the Self-managed Open Source/Commercial Software tab. Then, click Docker Stdout and Stderr - Container.
In the Select Logstore step of the Import Data wizard, select the project and the Logstore. Then, click Next.
In the Create Machine Group step of the Import Data wizard, click Create Machine Group. In the panel that appears, set the Machine Group Identifier parameter to Custom Identifier and enter log-docker-demo in the Custom Identifier field. Then, click OK. The custom identifier that you enter in the Custom Identifier field must be the same as the custom identifier that you specify for the ${your_machine_group_user_defined_id} parameter in Step 1.2.
In the Logtail Configuration step of the Import Data wizard, configure settings in the Global Configurations, Input Configurations, and Processor Configurations sections and click Next. For more information about the parameters, see Collect text logs from servers.
In the Query and Analysis Configurations step of the Import Data wizard, create indexes. For more information, see Create indexes. Then, the system collects data based on the created Logtail configuration.

3. View the collected data

3.1 View the stdout and stderr of an application container

Run the docker logs ${container_ID} command to view the stdout and stderr of an application container. In this example, the aliweb application container is used.

You can also view the stdout and stderr of an application container in a log file. By default, Docker stores the stdout and stderr of an application container in the /var/lib/docker/containers directory of the host on which the application container runs. Each application container corresponds to a folder named after the container ID, and the container logs are stored in this folder. The name of a file in the folder is ${container_id}-json.log.

3.2 View data in the Simple Log Service console

By default, each collected stdout or stderr contains the following fields.

Field	Description
__source__	The IP address of the Logtail container.
__tag__:__hostname__	The name of the host.
__tag__:__receive_time__	The time when Simple Log Service receives the log.
_time_	The time when the log is uploaded to Simple Log Service. Example: `2024-02-02T02:18:41.979147844Z`.
_source_	The type of the source. Valid values: stdout and stderr.
_image_name_	The name of the image.
_container_name_	The name of the application container.
_container_ip_	The IP address of the application container.

References

For more information about the status of Logtail, see View the status of Logtail.
For more information about how to use Docker, see Install and use Docker on a Linux instance.
For more information about how to collect text logs from Docker containers, see Collect logs from Docker containers.
For more information about how to collect text logs from a host on which your application container runs, see Collect text logs from servers. By default, the root directory of the host is mounted on the /logtail_host directory of the Logtail container.
After logs are collected to a Logstore, you can create indexes for the logs. For more information, see Create indexes. Then, you can query and analyze the collected logs. For more information, see Query and analyze logs.
For more information about how to troubleshoot the errors that occur when you use Logtail to collect Docker container logs, see What do I do if errors occur when I collect logs from containers?