This topic describes the limits of Logtail, including the limits on supported environments, log files, containers, checkpoints, Logtail configurations, machine groups, performance metrics, and troubleshooting.
Supported environments
Item | Description |
Architecture |
|
Memory | If no workloads are running and no plug-ins are enabled, a minimum of 20 MB of memory is required. If no workloads are running and at least one plug-in is enabled, a minimum of 120 MB of memory is required. The actual usage of memory varies based on the collection rate, monitored directories, number of log files, and number of synchronously sent requests. |
Operating system |
|
Kubernetes |
|
Docker | The collection of stdout and stderr from containers has the following limits:
|
The logtail-ds component for Container Service for Kubernetes (ACK) |
Important The preceding components belong to the system-cluster-critical class. If cluster resources are insufficient, we recommend that you do not deploy the components. Otherwise, the existing pods on the node may be evicted. |
Limits on log files
Item | Description |
Size of a single log | By default, the maximum size of a log is 512 KB. You can change the value of the startup parameter max_read_buffer_size to change the size. The maximum size of a log cannot exceed 8 MB. For more information, see Configure the startup parameters of Logtail. If a multi-line log is split based on a regular expression that is used to match the beginning of the first line of a log, the maximum size of each log after splitting is still 512 KB. If the size of a log exceeds 512 KB, the log is forcefully split into multiple logs for collection. For example, if the size of a log is 1,025 KB, the log is split into logs of the following sizes: 512 KB, 512 KB, and 1 KB. Then, the logs are collected in sequence and considered incomplete logs. |
Log file encoding | Logtail supports log files that are encoded in UTF-8 and GBK. We recommend that you use UTF-8-encoded log files to improve processing performance. Warning If log files are encoded in other formats, issues such as garbled characters and data loss may occur. |
Size of a log file | Unlimited. |
Log file rotation | By default, the maximum number of log files in a rotation queue is 20. You can change the value of the startup parameter logreader_max_rotate_queue_size to change the number. For more information, see Configure the startup parameters of Logtail. You can specify a log path in the Important Make sure that the two formats do not exist at the same time in a Logtail instance. If the two formats exist at the same time, the logs in a log file may be collected by using multiple Logtail configurations, and duplicate data may be collected. If more than 20 log files are not processed, new logs will be lost. In this case, you must check whether the write quota of shards exceeds the limit and adjust concurrency-related parameters. For more information, see Recommended parameter values. |
Log collection behavior performed when log parsing is blocked | When log parsing is blocked, Logtail keeps the descriptor of the log file open to prevent the log file from being deleted during the blocking period and log loss. If the log file is rotated multiple times during the blocking period, Logtail puts the log file into a rotation queue. |
Regular expression | Logtail uses regular expressions that are compatible with Perl. |
JSON | Standard JSON formats defined in RFC 7159 and ECMA-404 are supported. Non-standard JSON formats, such as |
Multiple Logtail configurations for one log file | By default, you can use only one Logtail configuration to collect logs from a log file. For more information about how to use multiple Logtail configurations to collect logs from a log file, see How do I collect multiple copies of logs in a file? Important When you use multiple Logtail configurations to collect logs from a log file, the read I/O, computing resources, and network I/O increase accordingly. |
File opening behavior | Logtail keeps the log files from which you want to collect logs and the log files in a rotation queue open to ensure the integrity of collected data. A log file is closed in the following scenarios:
If you want to release the file handle within a specified period of time after a log file is deleted, regardless of whether log collection from the log file is complete or whether new logs are still written to the log file, you can configure the force_release_deleted_file_fd_timeout parameter to specify a timeout period. For more information, see Configure the startup parameters of Logtail. |
First log collection behavior | Logtail collects data only from incremental log files. If the size of a log file exceeds the limit of 1 MB the first time the modification to the log file is detected, Logtail collects data from the last 1 MB. If the size of the log file does not exceed 1 MB, Logtail collects data from the beginning of the log file. The limit for container stdout and stderr is 512 KB. You can change the value of the tail_size_kb parameter in a Logtail configuration to change the limit. For more information, see Logtail configurations. If a log file is not modified after a Logtail configuration is delivered, Logtail does not collect data from the log file. For more information about how to collect logs from historical log files, see Import historical logs from log files. |
File overwriting behavior | Logtail uses an inode and the hash value of the first 1,024 bytes of a log file to identify the log file. If a log file is overwritten and the inode or the hash value of the first 1,024 bytes of the log file changes, the log file is considered a log file from which logs are not collected, and the logs are collected from the beginning of the log file. If the inode or the hash value does not change, the logs in the log file are not collected. |
File transfer behavior | If a log file is transferred and the matched Logtail configuration is not used to collect logs from the log file before the log file is transferred, the log file is considered a log file from which logs are not collected, and the logs are collected from the beginning of the log file. In this scenario, if the matched Logtail configuration is used to collect logs from the log file, the logs in the log file are not collected. |
File collection history | Logtail retains the historical collection progress of historical log files in the memory to ensure that only incremental data is collected after the log files are changed. If the historical collection progress of a historical log file is retained longer than a specified period and new data is written to the log file, duplicate data is collected.
|
Non-standard text logs | If a log contains \0, the log is truncated at the first occurrence of \0, and the rest of the log is discarded. For other escape characters such as ASCII codes that are used to represent colors or non-printable characters, Logtail directly sends the characters. |
Limits on containers
Both the limits on log files and the limits on containers apply when you use Logtail to collect container logs.
Item | Description |
First log collection behavior | When you collect container stdout and stderr and if the size of a log file exceeds the limit of 512 KB the first time the modification to the log file is detected, Logtail collects data from the last 512 KB. If the size of the log file does not exceed 512 KB, Logtail collects data from the beginning of the log file. You can change the value of the StartLogMaxOffset parameter in a Logtail configuration to change the limit. For more information, see Use the Simple Log Service console to collect container stdout and stderr in DaemonSet mode. |
Symbolic link | When you collect logs from container files, files and directories of files cannot be symbolic links. |
Container lifecycle | By default, Logtail can collect logs from a container only if the container lifecycle lasts 10 seconds or longer. When you collect logs from a container file, Logtail limits the number of log updates in the container file to 10 within a 3-minute period to ensure collection performance. You can change the values of the startup parameters docker_config_update_interval and max_docker_config_update_times to change the settings. For more information, see Configure the startup parameters of Logtail. |
File rotation for stdout and stderr | Container stdout and stderr files are rotated by Docker or kubelet. By default, the size of stdout and stderr files that are rotated by kubelet is 10 MB, and the size of stdout and stderr files that are rotated by Docker is 100 MB. If the output rate of container stdout and stderr is greater than 10 MB/s, the stdout and stderr files are rotated at a higher speed. In this case, we recommend that you collect logs from container files or change the value of the containerLogMaxSize parameter to prevent log loss. |
Logging driver for stdout and stderr | If you use Docker as a container runtime, you must add |
Limits on checkpoints
Item | Description |
Checkpoint timeout period | In default scenarios, if a log file is not modified within 30 days, the checkpoint of the log file is automatically deleted. If |
Checkpoint storage policy | Checkpoints are automatically stored at intervals of 15 minutes and at the point in time when Logtail exits. You can change the value of the startup parameter check_point_dump_interval to change the checkpoint storage policy. For more information, see Configure the startup parameters of Logtail. |
Checkpoint storage path | By default, checkpoints are stored in the |
Handling during downtime | Checkpoints are saved at regular intervals. If downtime occurs, data collection resumes from the last completely saved checkpoint. This may cause duplicate data collection. You can change the checkpoint storage policy to prevent duplicate data collection. |
Limits on Logtail configurations
Item | Description |
Latency for configuration updates to take effect | Updates to Logtail configurations that are performed by using the Simple Log Service console or by calling an API operation require approximately 30 seconds to take effect. |
Dynamic loading of Logtail configurations | Logtail configurations can be dynamically loaded. An update to a Logtail configuration does not affect other Logtail configurations. |
Number of Logtail configurations that can be dynamically loaded for a single Logtail instance | Unlimited. However, we recommend that you load no more than 100 Logtail configurations for a server. |
Log generation by using a third-party flusher | If a Logtail configuration is created in the Simple Log Service console or by calling an API operation, the Logtail configuration is associated with a Logstore. Therefore, when you configure a third-party flusher in your plug-in configuration, Logtail automatically sends a copy of data to the Logstore. |
Multi-account and cross-account | Multi-account and cross-account log collection is supported. For more information, see Multiple Alibaba Cloud accounts and Use Logtail to collect container logs across Alibaba Cloud accounts. |
Multi-region | By default, multi-region log collection is not supported. If you need to collect logs from multiple regions, submit a ticket. |
Global acceleration | Global acceleration is supported. After you enable global acceleration on the Simple Log Service side, you must configure global acceleration on the Logtail side. For more information, see Log collection acceleration. |
Limits on machine groups
Item | Description |
Number of machines | Unlimited. However, we recommend that you configure no more than 100,000 machines. Otherwise, heartbeats cannot be obtained. |
Number of Logtail configurations that can be applied | Unlimited. However, we recommend that you apply no more than 1,000 Logtail configurations. |
Limits on performance metrics
Item | Description |
Throughput for log processing | The default transmission speed of raw logs is limited to 20 MB/s. Log data is uploaded after it is encoded and compressed. The compression ratio ranges from 5:1 to 10:1. If the transmission speed is faster than the default value, logs may be lost. You can change the value of the startup parameter max_bytes_per_sec to change the transmission speed. For more information, see Configure the startup parameters of Logtail. |
Maximum processing speed for logs | Single-core-enabled processing speed:
You can configure the startup parameter process_thread_count to configure multiple threads. This helps improve performance by 150% to 300%. |
Maximum numbers of monitored directories and files | The maximum numbers of monitored directories and files are related to the mem_usage_limit parameter. The default value of the mem_usage_limit parameter is 384 MB in a host environment and 2,048 MB in a container environment. The following categories are used:
If the number for one of the preceding categories reaches the upper limit, Logtail no longer monitors the rest of the directories and files that correspond to the category. You can narrow the scope of monitored directories in a Logtail configuration or change the value of the startup parameter mem_usage_limit to increase the number of monitored directories. For more information about the mem_usage_limit parameter, see Configure the startup parameters of Logtail. Logtail installed on a Linux server allows you to use the inotify mechanism to monitor directories. This helps shorten the latency of log collection. The maximum number of directories, including subdirectories, which can be monitored by using the inotify mechanism is 3,000. |
Policy used to process excessive resource consumption | If the amount of resources occupied by Logtail remains higher than the upper limit for 5 minutes, Logtail is forcefully restarted. The restart may cause data loss or duplication. |
Multi-tenant isolation | Logtail configurations are isolated. If an error occurs in a Logtail configuration, other Logtail configurations are not affected. |
Log collection latency | In normal cases, Logtail can collect a log less than 1 second after the log is written to disk. |
Log upload policy | Before Logtail uploads logs, Logtail aggregates the logs in the same file. Logtail starts to upload logs when the number of logs exceeds 4,000, the total size of logs exceeds 512 KB, or the log collection duration exceeds 3 seconds. |
Limits on troubleshooting
Item | Description |
Network error handling | If a network error occurs, Logtail automatically retries the data collection task and adjusts the retry interval. In extreme cases, logs may be repeatedly collected or discarded due to the following issues:
|
Processing of threshold-crossing events | If a data transmission speed exceeds the upper limit of a Logstore, Logtail blocks log collection and automatically retries the data collection task. We recommend that you increase the number of shards for the Logstore. |
Time errors of Logtail | If retries fail five times because the time difference between the request time and the response time is larger than 15 minutes, Logtail discards related data. A maximum of five retries are allowed. We recommend that you correct the time of your machine where Logtail resides. |
Non-existence of a specified project or Logstore | If retries fail five times, Logtail discards related data. A maximum of five retries are allowed. The failure may occur if you deleted your Logstore by calling an API operation. We recommend that you delete the Logtail configurations and disassociate the Logtail configurations from your machine groups by calling API operations. |
Failed authentication | If retries fail five times, Logtail discards related data. A maximum of five retries are allowed. The failure may occur in the following scenarios:
|
Other unknown errors | If retries fail five times, Logtail discards related data. A maximum of five retries are allowed. |
Maximum retry period before timeout | If data fails to be transmitted and the issue lasts for more than 6 hours, Logtail discards the data. |
Status self-check | If an exception occurs, Logtail restarts. For example, if an application unexpectedly exits or the resource usage exceeds the specified upper limit, Logtail restarts. |