All Products
Search

This Product

    :Logtail

    Document Center

    :Logtail

    Last Updated:Nov 06, 2024

    Logtail is a log collection agent that is provided by Simple Log Service. You can use Logtail to collect logs from multiple data sources, including Alibaba Cloud Elastic Compute Service (ECS) instances, servers in data centers, and servers from third-party cloud service providers. This topic describes the features, benefits, limits, and configuration process of Logtail.

    Configuration process

    image

    1. Install Logtail on a server.

    2. Configure a user identifier for the server. This operation is required if your server is an ECS instance that belongs to a different Alibaba Cloud account, a server in a data center, or a server from a third-party cloud service provider.

      For more information, see Configure a user identifier.

    3. Create a machine group.

    4. Create a Logtail configuration and apply the Logtail configuration to the machine group.

      You can perform the preceding operations in the Simple Log Service console. For more information, see collect text logs and collect container logs.

    After you perform the preceding operations, Logtail collects logs from your server and sends the logs to a specified Logstore. You can query the logs by using the console, API, SDK, or CLI of Simple Log Service.

    Benefits

    • Supports non-intrusive log collection based on log files. You do not need to modify your application code, and log collection does not affect the operation of your applications.

    • Collects text logs, binary logs, HTTP logs, and container logs.

    • Collects logs from standard containers and various container clusters, such as Kubernetes clusters.

    • Handles exceptions that occur in the log collection process. If issues such as network or server exceptions occur, Logtail retries log collection and caches data locally to ensure data security.

    • Provides centralized management based on Simple Log Service. After you install Logtail on a server from which you want to collect logs and create a machine group and Logtail configuration, Logtail collects logs from the server.

    • Provides a comprehensive self-protection mechanism. To ensure that Logtail does not significantly affect the performance of other services that run on the same server as Logtail, Simple Log Service limits the CPU, memory, and network resources that can be used by Logtail and provides a self-protection mechanism.

    Processing capabilities and limits

    For more information, see Logtail limits.

    Terms

    • Machine group: A machine group contains one or more servers from which a specific type of logs are collected. After you apply a Logtail configuration to a machine group, Simple Log Service collects logs from all servers in the machine group based on the Logtail configuration.

      Simple Log Service uses machine groups to manage all servers from which you want to collect logs by using Logtail. You can define a machine group based on an IP address or a custom identifier. You can manage machine groups in the Simple Log Service console. For example, you can create or delete a machine group and add a server to or remove a server from a machine group. For more information, see Overview.

    • Logtail: Logtail is a log collection agent that is provided by Simple Log Service. Logtail runs on servers from which you want to collect logs.

      • Linux: In Linux, Logtail is installed in the /usr/local/ilogtail directory and initiates two independent processes whose names start with ilogtail. One is a collection process and the other is a daemon. The program operational logs are stored in the /usr/local/ilogtail/ilogtail.LOG file. For more information, see Install Logtail on a Linux server.

      • Windows:

        • Logtail (32-bit)

          • In 32-bit Windows, Logtail is installed in the C:\Program Files\Alibaba\Logtail directory.

          • In 64-bit Windows, Logtail is installed in the C:\Program Files (x86)\Alibaba\Logtail directory.

            Note

            You can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.

        • Logtail (64-bit)

          You can install Logtail (64-bit) only in 64-bit Windows. The installation directory is C:\Program Files\Alibaba\Logtail.

        To check the status of Logtail, you can perform the following operations: Choose Control Panel > Administrative Tools > Services. If you install Logtail V1.0.0.0 or later, view the LogtailDaemon service. If you install Logtail V0.x.x.x, view the LogtailWorker service. The program operational logs are stored in the ilogtail.LOG file of the installation directory. For more information, see Install Logtail on a Windows server.

    • Logtail configuration: A Logtail configuration is a set of policies used by Logtail to collect logs. You can specify the data source and collection mode to create custom Logtail configurations for log collection. A Logtail configuration is used to collect a specific type of logs from servers, parse the collected logs, and send the logs to a specified Logstore of Simple Log Service.

    Features

    Feature

    Description

    Real-time log collection

    Logtail dynamically monitors log files and reads and parses incremental logs in real time. In most cases, logs are sent to Simple Log Service within 3 seconds after the logs are generated. For more information, see Log collection process of Logtail.

    Note

    Logtail does not collect historical logs. Logs that are read 12 hours or later after the logs are generated are discarded. For more information about how to collect logs from historical log files, see Import historical logs from log files.

    Automatic log rotation

    Multiple applications rotate log files based on the file size or date. In the rotation process, original log files are renamed and new empty log files are created. For example, files such as app.LOG.1 and app.LOG.2 are generated for the app.LOG file after log rotation. You can specify the file to which collected logs are written. Example: app.LOG. Logtail automatically monitors the log rotation process and ensures that no logs are lost during this process.

    Support for multiple data sources

    Logtail can collect text logs, syslogs, HTTP logs, and MySQL binary logs. For more information, see Data collection overview.

    Compatibility with an open-source collection agent

    Logtail can collect data that is collected by using open-source software such as Logstash and Beats to Simple Log Service. For more information, see Data collection overview.

    Automatic handling of collection exceptions

    If data transmission fails due to an exception such as Simple Log Service errors, network errors, or quota exhaustion, Logtail actively retries log collection based on the specific scenario. If the retry fails, Logtail writes the data to its local cache and sends the data again after 3 seconds. For more information, see How do I use the automatic diagnostic tool of Logtail?

    Flexible collection configuration

    You can collect logs in a flexible manner based on Logtail configurations. You can specify the directories and files from which logs are collected. Exact match and wildcard match are supported. You can specify the log collection mode and the fields that you want to extract. You can use a regular expression to extract logs.

    The log data models of Simple Log Service require that each log have a precise timestamp. Logtail supports custom log time formats, which allows you to extract the required timestamp information from log data of different formats.

    Automatic synchronization of Logtail configurations

    After you create or update a Logtail configuration in the Simple Log Service console, Logtail automatically receives and applies the configuration within 3 minutes in most cases. No logs are lost during the Logtail update process.

    Status monitoring

    Logtail monitors its CPU and memory consumption in real time. This helps prevent Logtail from consuming excessive resources. The overconsumption of resources may affect other services that run on the same server as Logtail. If the resource usage of Logtail exceeds the limit, Logtail automatically restarts. If the network bandwidth usage exceeds the limit, Logtail triggers throttling. For more information, see Startup configuration file (ilogtail_config.json).

    Transmission of signed data

    To prevent data from being tampered with during data transmission, Logtail obtains a private token from Simple Log Service over a trusted channel and signs all log data packets that are sent.

    Note

    Logtail obtains a private token over HTTPS to ensure the security of your token.

    Data collection reliability

    During data collection, Logtail stores the collected checkpoint information to a local server on a regular basis. If an exception such as unexpected shutdown of a server occurs or a process unexpectedly exits, Logtail collects data from the last recorded checkpoint after it is restarted. This prevents data loss. Logtail runs based on the startup parameters that are specified in the startup configuration file. If the resource usage of Logtail exceeds a limit for more than 5 minutes, Logtail is forcefully restarted. Duplicate data may be collected after the restart.

    Logtail uses internal mechanisms to improve log collection reliability. However, logs may be lost in the following situations:

    • Logtail is not running, but log files are rotated multiple times.

    • The rotation rate of log files is extremely high, such as one rotation per second.

    • The log collection rate is lower than the log generation rate for a long period of time.

    References