Simple Log Service:Collect text logs from a Docker container

Overview

You can use a Logtail container to collect text logs in various formats from multiple application containers on a host. You need to only create Logtail configurations for different Logstores and deliver the Logtail configurations to the Logtail container. This way, the Logtail container sends the collected text logs to the corresponding Logstores based on the specified configurations. This topic describes how to collect text logs from an NGINX container by performing the following steps:

• Step 1: Install a Logtail container and create a machine group

• Step 2: Create a Logtail configuration

If you are familiar with log collection by using Logtail, you can directly perform operations in the console as prompted.

Step 1: Install a Logtail container and create a machine group

1. Pull a Logtail image.

   Log on to the host. Obtain the value of the ${region_id} parameter based on the region in which your Simple Log Service project resides. Replace the ${region_id} parameter with the actual region ID and run the following command to pull a Logtail image:

   Important

   • For more information about the ID of each region, see Supported regions. For example, if your SLS project resides in the China (Hangzhou) region, you must replace the ${region_id} parameter with cn-hangzhou.

   • If your server resides in a virtual private cloud (VPC), you must replace registry with registry-vpc.

   docker pull registry.${region_id}.aliyuncs.com/log-service/logtail

2. Start a Logtail container.

   1. Parameters

      Parameter	Description
      ${region_name}	The ID of the region in which your Simple Log Service project resides and the network type used by the project. For more information about the ID of each region, see Supported regions. For more information about how to select a network type, see Select a network type. If your project uses the Internet, specify the value in the ${region_id}-internet format. For example, if your project resides in the China (Hangzhou) region, set the value to cn-hangzhou-internet. If your project uses an Alibaba Cloud internal network, specify the value in the ${region_id} format. For example, if your project resides in the China (Hangzhou) region, set the value to cn-hangzhou.
      ${aliyun_account_id}	The ID of the Alibaba Cloud account to which your Simple Log Service project belongs. For more information about how to obtain the ID, see the "Obtain the ID of the Alibaba Cloud account to which your Simple Log Service project belongs" section of the Configure a user identifier topic.
      ${user_defined_id}	The custom identifier of your machine group. Example: user-defined-docker-1. The custom identifier must be unique in the region in which your project resides.

   2. Replace the ${region_name}, ${aliyun_account_id}, and ${user_defined_id} parameters in the following command based on the parameter description and run the following command to start the Logtail container:

      docker run -d \
          -v /:/logtail_host:ro \
          -v /var/run/docker.sock:/var/run/docker.sock \
          --env ALIYUN_LOGTAIL_CONFIG=/etc/ilogtail/conf/${region_name}/ilogtail_config.json \
          --env ALIYUN_LOGTAIL_USER_ID=${aliyun_account_id} \
          --env ALIYUN_LOGTAIL_USER_DEFINED_ID=${user_defined_id} \
          registry.${your_region_name}.aliyuncs.com/log-service/logtail

      Important

      You can configure custom settings for the startup parameters of the Logtail container only if the following conditions are met:

      • The following environment variables are configured: ALIYUN_LOGTAIL_CONFIG, ALIYUN_LOGTAIL_USER_ID, and ALIYUN_LOGTAIL_USER_DEFINED_ID.

      • The /var/run directory of the host is mounted on the /var/run directory of the Logtail container.

      • The root directory of the host is mounted on the /logtail_host directory of the Logtail container.

      • If the The parameter is invalid : uuid=none error message is contained in the /usr/local/ilogtail/ilogtail.LOG log file, you must create a file named product_uuid on the host, enter a valid UUID in the created file, such as 169E98C9-ABC0-4A92-B1D2-AA6239C0D261, and then mount the file on the /sys/class/dmi/id/product_uuid directory of the Logtail container.

3. Create a custom identifier-based machine group.

   1. Log on to the Simple Log Service console. In the Projects section, click the project that you want to manage.

   2. In the left-side navigation pane, choose Resources > Machine Groups. In the Machine Groups list, choose > Create Machine Group.

   3. In the Create Machine Group panel, configure parameters and click OK. The following table describes the parameters.

      Parameter	Description
      Name	The name of the machine group. The name must meet the following requirements: The name can contain only lowercase letters, digits, hyphens (-), and underscores (_). The name must start and end with a lowercase letter or a digit. The name must be 2 to 128 characters in length. Important After you create a machine group, you cannot change the name of the machine group. Proceed with caution.
      Machine Group Identifier	The identifier type of the machine group. In this example, Custom Identifier is selected.
      Machine Group Topic	Optional. The topic of the machine group. The topic is used to identify the logs that are generated by different servers. For more information, see Log topics.
      Custom Identifier	The specified custom identifier, which is the value of the ${user_defined_id} parameter when you start the Logtail container.

4. Check the status of the machine group.

   In the Machine Groups section, find the machine group that you created and click its name. On the Machine Group Configurations page, view the configurations of the machine group and the status of the server.

   • If the value in the Heartbeat column for a server is OK, Logtail on the server is connected to Simple Log Service.

   • If the value in the Heartbeat column for a server is FAIL, an error occurs. For more information about how to troubleshoot the error, see How do I troubleshoot an error related to a Logtail machine group in a host environment?

     Important

     Approximately 2 minutes are required to create a machine group. If the machine group is not created, the heartbeat status of the machine group is FAIL. In this case, click Refresh 2 minutes later.

Step 2: Create a Logtail configuration

1. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to manage.

2. Click Logtail Configurations. On the Logtail Configuration page, click Add Logtail Configuration.

3. In the Quick Data Import dialog box, click Docker File - Container.

4. In the Machine Group Configuration step, select the machine group that you created in Step 1 to add the machine group to the Applied Server Groups section. Click Next.

5. In the Logtail Configuration step, configure the Configuration Name and File Path parameters and use the default values for other parameters. For more information about the parameters, see Collect text logs from servers. You can also add Logtail plug-ins to process text logs. For more information, see Overview of Logtail plug-ins for data processing. Click Next.

   Note

   • Configuration Name: the name of the Logtail configuration. The name can contain only lowercase letters, digits, hyphens (-), and underscores (_). It must start and end with a lowercase letter or a digit.

   • File Path: the absolute path of the log file in the application container. Example: /var/log/nginx/access.log.

   

6. In the Query and Analysis Configurations step, click Automatic Index Generation and then click Refresh. This way, you can preview the collected logs.

Reserved fields

By default, each collected log contains specific fields. The following table describes the fields.

Related operations

View the status of Logtail

You can run the docker exec ${logtail_container_id} /etc/init.d/ilogtaild status command to view the status of Logtail.

View the version number, IP address, and startup time of Logtail

You can run the docker exec ${logtail_container_id} cat /usr/local/ilogtail/app_info.json command to view the information about Logtail.

View the operational logs of Logtail

[sudo@iZb****exh2Z ilogtail]# docker exec a287de895e40 tail -n 5 /usr/local/ilogtail/ilogtail.LOG
[2018-02-06 08:13:35.721864]    [INFO]    [8]    [build/release64/sls/ilogtail/LogtailPlugin.cpp:104]    logtail plugin Resume:start
[2018-02-06 08:13:35.722135]    [INFO]    [8]    [build/release64/sls/ilogtail/LogtailPlugin.cpp:106]    logtail plugin Resume:success
[2018-02-06 08:13:35.722149]    [INFO]    [8]    [build/release64/sls/ilogtail/EventDispatcher.cpp:369]    start add existed check point events, size:0
[2018-02-06 08:13:35.722155]    [INFO]    [8]    [build/release64/sls/ilogtail/EventDispatcher.cpp:511]    add existed check point events, size:0    cache size:0    event size:0    success count:0
[2018-02-06 08:13:39.725417]    [INFO]    [8]    [build/release64/sls/ilogtail/ConfigManager.cpp:3776]    check container path update flag:0    size:1

start umount useless mount points, /shm$|/merged$|/mqueue$
umount: /logtail_host/var/lib/docker/overlay2/3fd0043af174cb0273c3c7869500fbe2bdb95d13b1e110172ef57fe840c82155/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/d5b10aa19399992755de1f85d25009528daa749c1bf8c16edff44beab6e69718/merged: must be superuser to unmount
umount: /logtail_host/var/lib/docker/overlay2/5c3125daddacedec29df72ad0c52fac800cd56c6e880dc4e8a640b1e16c22dbe/merged: must be superuser to unmount
......
xargs: umount: exited with status 255; aborting
umount done
start logtail
ilogtail is running
logtail status:
ilogtail is running

Restart Logtail

Sample code to restart Logtail:

[sudo@iZb****xh2Z ilogtail]# docker exec a287de895e40 /etc/init.d/ilogtaild stop
kill process Name: ilogtail pid: 7
kill process Name: ilogtail pid: 8
stop success
[sudo@iZb****xh2Z ilogtail]# docker exec a287de895e40 /etc/init.d/ilogtaild start
ilogtail is running

References

• For more information about how to use Docker, see Install Docker.

• For more information about how to collect stdout and stderr from Docker containers, see Collect stdout and stderr from Docker containers.

• For more information about how to collect text logs from a host on which your application container runs, see Collect text logs from servers. By default, the root directory of the host is mounted on the /logtail_host directory of the Logtail container.

• After logs are collected to a Logstore, you can create indexes for the logs. For more information, see Create indexes. Then, you can query and analyze the collected logs. For more information, see Query and analyze logs.

• For more information about how to troubleshoot the errors that occur when you use Logtail to collect logs from Docker containers, see What do I do if errors occur when I collect logs from containers?