Item | Description |
Size of a single log | By default, the maximum size of a log is 512 KB. You can change the value of the startup parameter max_read_buffer_size to change the size. The maximum size of a log cannot exceed 8 MB. For more information, see Configure the startup parameters of Logtail. If a multi-line log is split based on a regular expression that is used to match the beginning of the first line of a log, the maximum size of each log after splitting is still 512 KB. If the size of a log exceeds 512 KB, the log is forcefully split into multiple logs for collection. For example, if the size of a log is 1,025 KB, the log is split into logs of the following sizes: 512 KB, 512 KB, and 1 KB. Then, the logs are collected in sequence and considered incomplete logs. |
Log file encoding | Logtail supports log files that are encoded in UTF-8 and GBK. We recommend that you use UTF-8-encoded log files to improve processing performance. Warning If log files are encoded in other formats, issues such as garbled characters and data loss may occur. |
Size of a log file | Unlimited. |
Log file rotation | By default, the maximum number of log files in a rotation queue is 20. You can change the value of the startup parameter logreader_max_rotate_queue_size to change the number. For more information, see Configure the startup parameters of Logtail. You can specify a log path in the xxx.log or xxx.log* format. Important Make sure that the two formats do not exist at the same time in a Logtail instance. If the two formats exist at the same time, the logs in a log file may be collected by using multiple Logtail configurations, and duplicate data may be collected. If more than 20 log files are not processed, new logs will be lost. In this case, you must check whether the write quota of shards exceeds the limit and adjust concurrency-related parameters. For more information, see Recommended parameter values. |
Log collection behavior performed when log parsing is blocked | When log parsing is blocked, Logtail keeps the descriptor of the log file open to prevent the log file from being deleted during the blocking period and log loss. If the log file is rotated multiple times during the blocking period, Logtail puts the log file into a rotation queue. |
Regular expression | Logtail uses regular expressions that are compatible with Perl. |
JSON | Standard JSON formats defined in RFC 7159 and ECMA-404 are supported. Non-standard JSON formats, such as {"name": "\xE5\xAD\xA6"} , are not supported. |
Multiple Logtail configurations for one log file | By default, you can use only one Logtail configuration to collect logs from a log file. For more information about how to use multiple Logtail configurations to collect logs from a log file, see How do I collect multiple copies of logs in a file? Important When you use multiple Logtail configurations to collect logs from a log file, the read I/O, computing resources, and network I/O increase accordingly. |
File opening behavior | Logtail keeps the log files from which you want to collect logs and the log files in a rotation queue open to ensure the integrity of collected data. A log file is closed in the following scenarios: The log file is not modified within 5 minutes. The log file is rotated, and all logs in the log file are collected. The Logtail configuration is updated.
If you want to release the file handle within a specified period of time after a log file is deleted, regardless of whether log collection from the log file is complete or whether new logs are still written to the log file, you can configure the force_release_deleted_file_fd_timeout parameter to specify a timeout period. For more information, see Configure the startup parameters of Logtail. |
First log collection behavior | Logtail collects data only from incremental log files. If the size of a log file exceeds the limit of 1 MB the first time the modification to the log file is detected, Logtail collects data from the last 1 MB. If the size of the log file does not exceed 1 MB, Logtail collects data from the beginning of the log file. The limit for container stdout and stderr is 512 KB. You can change the value of the tail_size_kb parameter in a Logtail configuration to change the limit. For more information, see Logtail configurations. If a log file is not modified after a Logtail configuration is delivered, Logtail does not collect data from the log file. For more information about how to collect logs from historical log files, see Import historical logs from log files. |
File overwriting behavior | Logtail uses an inode and the hash value of the first 1,024 bytes of a log file to identify the log file. If a log file is overwritten and the inode or the hash value of the first 1,024 bytes of the log file changes, the log file is considered a log file from which logs are not collected, and the logs are collected from the beginning of the log file. If the inode or the hash value does not change, the logs in the log file are not collected. |
File transfer behavior | If a log file is transferred and the matched Logtail configuration is not used to collect logs from the log file before the log file is transferred, the log file is considered a log file from which logs are not collected, and the logs are collected from the beginning of the log file. In this scenario, if the matched Logtail configuration is used to collect logs from the log file, the logs in the log file are not collected. |
File collection history | Logtail retains the historical collection progress of historical log files in the memory to ensure that only incremental data is collected after the log files are changed. If the historical collection progress of a historical log file is retained longer than a specified period and new data is written to the log file, duplicate data is collected. By default, the historical collection progress of historical log files is retained for up to one month. If the number of historical log files in the same directory exceeds 5,000, the historical collection progress of the log files within the previous week is retained. If the number of historical log files in the same directory exceeds 10,000, the historical collection progress of the log files within the previous day is retained.
|
Non-standard text logs | If a log contains \0, the log is truncated at the first occurrence of \0, and the rest of the log is discarded. For other escape characters such as ASCII codes that are used to represent colors or non-printable characters, Logtail directly sends the characters. |