Simple Log Service:Configure the startup parameters of Logtail

Scenarios

Recommended parameter values

If you want to collect logs from JSON files, use the following parameter values that are recommended based on previous experience. The collection performance of Logtail in full regex mode and in delimiter mode is similar to the collection performance of Logtail in JSON mode. The collection performance of Logtail in simple mode is five times higher than the collection performance of Logtail in JSON mode. Both the complexity of data and rules and the numbers of directories and files from which you want to collect logs affect CPU utilization and memory usage. We recommend that you configure the following parameters based on the values in the table and your business requirements.

• Host environment

  Parameter	Default collection speed	Collection speed higher than 10 MB/s	Collection speed higher than 20 MB/s	Collection speed higher than 40 MB/s
  cpu_usage_limit	0.4	1	2	4
  mem_usage_limit	384	1024	2048	4096
  max_bytes_per_sec	20971520	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	4	20	40	80

• Container or Kubernetes environment

  Environment variable	Default collection speed	Collection speed higher than 10 MB/s	Collection speed higher than 20 MB/s	Collection speed higher than 40 MB/s
  cpu_usage_limit	2	3	5	9
  mem_usage_limit	2048	2048	2048	4096
  max_bytes_per_sec	209715200	209715200	209715200	209715200
  process_thread_count	1	2	4	8
  send_request_concurrency	20	20	40	80
  resources.limits.cpu	500M	1000M	2000M	4000M
  resources.limits.memory	2 Gi	2 Gi	3 Gi	5 Gi

  The following section describes how to modify the startup parameters of Logtail in a container or Kubernetes environment:

  • If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in a Container Service for Kubernetes (ACK) cluster, we recommend that you modify the startup parameters of Logtail in the ACK console. On the Add-ons page, modify the parameters of the logtail-ds component.

  • If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in a self-managed container or Kubernetes cluster, you must modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration.

  • If the version of Logtail is earlier than Logtail-ds 1.7.3, you must modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration. You can also modify resources.limits.cpu and resources.limits.memory in daemonset > kube-system > logtail-ds to prevent the excessive usage of container resources.

If you configure the startup parameters of Logtail based on the values of the Collection speed higher than 40 MB/s column in the preceding tables, the collection performance of Logtail approaches the upper limit. In this case, the performance does not significantly improve even if more threads are created. The following table describes the upper limit of the collection performance that Logtail can deliver in different collection modes.

Configure the startup parameters of Logtail

1. Open the /usr/local/ilogtail/ilogtail_config.json file on the server on which Logtail is installed.

   If you want to collect logs from a server, you can perform this step to configure the startup parameters of Logtail.

   If you want to collect logs from a container or a Kubernetes cluster, you can modify the startup parameters of Logtail by modifying DaemonSet-related environment variables. ConfigMaps are referenced by some environment variables, and the path to the ConfigMaps is configmap > kube-system > alibaba-log-configuration.

2. Configure the startup parameters of Logtail based on your business requirements.

   The following example shows the startup parameters of Logtail:

   {
       ...
       "cpu_usage_limit" : 0.4,
       "mem_usage_limit" : 384,
       "max_bytes_per_sec" : 20971520,
       "process_thread_count" : 1,
       "send_request_concurrency" : 4,
       "buffer_file_num" : 25,
       "buffer_file_size" : 20971520,
       "buffer_file_path" : "",
       ...
   }

   Note

   • The following table describes the common startup parameters of Logtail. You can use the default values for other startup parameters.

   • You can add or modify startup parameters based on your business requirements.

   Table 1. Startup parameters of Logtail

   Parameter	Type	Description	Example
   cpu_usage_limit	double	The CPU utilization threshold for Logtail. The calculation is based on a single core. Valid values: 0.1 to the number of CPU cores of the current server Default value: 0.4 Warning cpu_usage_limit specifies a soft limit. The actual CPU utilization of Logtail may exceed the limit. If the CPU utilization of Logtail remains higher than this limit for 5 minutes, the system triggers a circuit breaker. Then, Logtail automatically restarts. For example, if you set the parameter to 0.4 and the CPU utilization of Logtail remains higher than 40% for 5 minutes based on a single core, Logtail automatically restarts. In most cases, a single core supports a collection speed of approximately 100 MB/s in simple mode and approximately 20 MB/s in full regex mode. Note If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you specify a CPU utilization threshold in the ACK console. On the Add-ons page, modify the LogtailDSLimitCPU parameter of the logtail-ds component. For more information, see Manage components.	"cpu_usage_limit" : 0.4
   mem_usage_limit	int	The memory usage threshold for Logtail. Unit: MB. Valid values: 128 to 8192 Default value: 384 for a host environment and 2048 for ACK components Warning mem_usage_limit specifies a soft limit. The actual memory usage of Logtail may exceed the limit. If the memory usage of Logtail remains higher than this limit for 5 minutes, the system triggers a circuit breaker. Then, Logtail automatically restarts. The collection speed, monitored directories, number of log files, and number of synchronously sent requests are related to the mem_usage_limit parameter. For more information, see Logtail limits. Note If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you specify a memory usage threshold in the ACK console. On the Add-ons page, modify the LogtailDSLimitMemory parameter of the logtail-ds component. For more information, see Manage components.	"mem_usage_limit" : 384
   max_bytes_per_sec	int	The highest speed at which Logtail sends raw data. Unit: bytes per second. Valid values: 1024 to 52428800 Default value: 20971520 Important If you set the parameter to a value that is greater than 20971520, the speed at which Logtail sends data is not limited. The value 20971520 indicates that the speed is 20 MB/s. For example, if you set the parameter to 2097152, Logtail can send data at a speed of up to 2 MB/s. Note If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you specify the maximum speed in the ACK console. On the Add-ons page, modify the LogtailDSMaxBytePerSec parametes of the logtail-ds component. For more information, see Manage components.	"max_bytes_per_sec" : 2097152
   process_thread_count	int	The number of threads that are used by Logtail to process data. Valid values: 1 to 64 Default value: 1 In most cases, a thread provides a write speed of 24 MB/s in simple mode and 12 MB/s in full regex mode. We recommend that you retain the default value for this parameter. Note If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you specify the number of threads in the ACK console. On the Add-ons page, modify the LogtailDSProcessThreadCount parameter of the logtail-ds component. For more information, see Manage components.	"process_thread_count" : 1
   send_request_concurrency	int	The number of concurrent requests that are sent by Logtail to asynchronously send data. Valid values: 1 to 1000 Default value: 20 If Simple Log Service provides a high transactions per second (TPS), you can set this parameter to a larger value. Each concurrent request supports a network throughput of 0.5 MB/s to 1 MB/s. The actual network throughput for a concurrent request varies based on the network latency. Note If the value of this parameter is large, concurrent requests may use an excessive number of network ports. In this case, you must change the values of the TCP-related parameters. If Logtail whose version is Logtail-ds 1.7.3 or later is deployed in an ACK cluster, we recommend that you specify the number of concurrent requests in the ACK console. On the Add-ons page, modify the LogtailDSSendRequestConcurrency parameter of the logtail-ds component. For more information, see Manage components.	"send_request_concurrency" : 4
   buffer_file_num	int	The maximum number of files that can be cached. Valid values: 1 to 100 Default value: 25 If a network error occurs or the amount of the written data exceeds the threshold that you specify, Logtail caches parsed logs to the local files in the installation directory. After the network is recovered and data is written within the threshold, Logtail retries to send the cached logs.	"buffer_file_num" : 25
   buffer_file_size	int	The maximum size of a cached file. Unit: bytes. Valid values: 1048576 to 104857600 Default value: 20971520 The maximum disk space that can be occupied by cached files is calculated by multiplying the value of the buffer_file_size parameter by the value of the buffer_file_num parameter.	"buffer_file_size" : 20971520
   buffer_file_path	String	The directory in which cached files are stored. This parameter is empty by default, which indicates that cached files are stored in the installation directory of Logtail. The default directory is /usr/local/ilogtail. If you specify a value for this parameter, you must move the cached files whose name matches logtail\_buffer\_file_* from the installation directory of Logtail to the directory that you specify. This way, Logtail can read, send, and then delete the cached files.	"buffer_file_path" : ""
   bind_interface	String	The name of the network interface controller (NIC) that is associated with the server on which Logtail is installed. This parameter is empty by default, which indicates that the server is automatically associated with an available NIC. If you specify a value for this parameter, such as eth1, Logtail uses the NIC to upload logs. This parameter is available only if Logtail runs on a Linux server.	"bind_interface" : ""
   check_point_filename	String	The path to the checkpoint files of Logtail. Default value: /tmp/logtail_check_point.	"check_point_filename" : /tmp/logtail_check_point
   check_point_dump_interval	int	The interval at which Logtail updates checkpoint files. Default value: 900. Unit: seconds. If you retain the default value, Logtail updates checkpoint files at 15-minute intervals. This parameter is available for only Linux Logtail V1.0.19 and later or Windows Logtail V1.0.19.0 and later.	"check_point_dump_interval" : 900
   user_config_file_path	String	The path to the file that stores Logtail configurations. The file is named user_log_config.json and stored in the directory of the BIN file that is created for the Logtail process.	"user_config_file_path" : user_log_config.json
   docker_file_cache_path	String	The path to the file that records the path mappings between container files and host files. By default, the path is /usr/local/ilogtail/docker_path_config.json. This parameter is available for only Linux Logtail V0.16.54 and later or Windows Logtail V0.16.54.0 and later.	"docker_file_cache_path": /usr/local/ilogtail/docker_path_config.json
   discard_old_data	Boolean	Specifies whether to discard historical logs. Default value: true. This value indicates that logs that were generated more than 12 hours before the current time are discarded.	"discard_old_data" : true
   ilogtail_discard_interval	int	The time threshold for discarding logs. If the difference between the time at which the logs were generated and the current time exceeds the threshold, the logs are discarded. Default value: 43200. Unit: seconds. The value 43200 indicates that the threshold is 12 hours.	"ilogtail_discard_interval": 43200
   working_ip	String	The server IP address that is reported by Logtail to Simple Log Service. This parameter is empty by default, which indicates that Simple Log Service automatically obtains the IP address of the server on which Logtail is installed.	"working_ip" : ""
   working_hostname	String	The server hostname that is reported by Logtail to Simple Log Service. This parameter is empty by default, which indicates that Simple Log Service automatically obtains the hostname of the server on which Logtail is installed.	"working_hostname" : ""
   max_read_buffer_size	long	The maximum size of a log that Logtail can read. Unit: bytes. Default value: 524288, which indicates that the maximum size is 512 KB. Maximum value: 8388608, which indicates that the maximum size is 8 MB. Unit: bytes. If the size of a log exceeds 524,288 bytes, you can change the value of this parameter.	"max_read_buffer_size" : 524288
   oas_connect_timeout	long	The timeout period of the connection that is established by Logtail to send a request to obtain the Logtail configuration or AccessKey pair. Default value: 5. Unit: seconds. If the connections cannot be established within the timeout period due to poor network conditions, you can change the value of this parameter.	"oas_connect_timeout" : 5
   oas_request_timeout	long	The timeout period of the request that is sent by Logtail to obtain the Logtail configuration or AccessKey pair. Default value: 10. Unit: seconds. If the connections cannot be established within the timeout period due to poor network conditions, you can change the value of this parameter.	"" : 10
   data_server_port	long	If you set the data_server_port parameter to 443, Logtail transfers data to Simple Log Service over HTTPS. This parameter is available for only Linux Logtail V1.0.10 and later or Windows Logtail V1.0.10.0 and later.	"data_server_port": 443
   enable_log_time_auto_adjust	Boolean	If you set the enable_log_time_auto_adjust parameter to true, the log time is adapted to the local time of the server. To ensure data security, Simple Log Service checks the time information in requests, including the time when the requests were sent by Logtail. This information indicates the time at which a request was sent. Simple Log Service rejects requests that are sent 15 minutes earlier or later than the time in Simple Log Service. The time information in a request is considered as the local time of the server. In some test scenarios, the local time must be changed to a future point in time. If you change the local time of the server, Simple Log Service rejects the requests from Logtail, and data cannot be written to Simple Log Service. You can use this parameter to adapt the log time to the local time of the server. This parameter is available for only Linux Logtail V1.0.19 and later or Windows Logtail V1.0.19.0 and later. Important If you set the enable_log_time_auto_adjust parameter to true, the offset between the time in Simple Log Service and the local time of the server is added to the log time. The offset is updated only if a request is rejected by Simple Log Service. Therefore, the time of a log that is queried by Simple Log Service may be different from the time at which the log is written. Part of the logic for Logtail changes based on the incremental increase of the system time. We recommend that you restart Logtail after you change the local time of the server.	"enable_log_time_auto_adjust": true
   accept_multi_config	Boolean	Specifies whether to allow Logtail to collect data from the same file by using multiple Logtail configurations. Default value: false. This value indicates that Logtail cannot collect data from the same file by using multiple Logtail configurations. By default, Logtail can use only one Logtail configuration to collect data from a file. If you want to allow Logtail to collect data from a file by using multiple Logtail configurations, you can set this parameter to true. Each Logtail configuration has an independent collection process. If multiple Logtail configurations are used to collect data from the same file, the CPU utilization and memory usage increase. This parameter is available for only Linux Logtail V0.16.26 and later or Windows Logtail V0.16.26.0 and later.	"accept_multi_config": true
   enable_checkpoint_sync_write	Boolean	Specifies whether to enable the sync write feature. Default value: false. This value indicates that the sync write feature is disabled. The sync write feature is used together with the ExactlyOnce write feature. After you enable the ExactlyOnce write feature, Logtail records fine-grained checkpoints by file to the disk of the server on which Logtail is installed. By default, Logtail does not call the sync function to write checkpoints to the disk. However, if buffered data fails to be written to the disk when the server restarts, the checkpoints may be lost. In this case, you can set the enable_checkpoint_sync_write parameter to true to enable the sync write feature. For more information, see Logtail configurations. This parameter is available for only Linux Logtail V1.0.20 and later or Windows Logtail V1.0.20.0 and later.	"enable_checkpoint_sync_write": false
   enable_env_ref_in_config	Boolean	Specifies whether to enable the environment variable replacement feature in Logtail configurations. Default value: false. After this feature is enabled, you can use ${xxx} as the placeholder for the environment variable xxx when you create a Logtail configuration in the Simple Log Service console. For example, if you set the log path to /${xxx}/logs and the environment variable to xxx=user, Logtail collects logs from the /user/logs path. If ${ and } are used in your Logtail configuration, you can use $${ and $} to escape the characters. This parameter is available for only Linux Logtail V1.0.31 and later or Windows Logtail V1.0.31.0 and later.	"enable_env_ref_in_config": false
   docker_config_update_interval	int	The minimum interval at which the container path is updated. This parameter is used together with the max_docker_config_update_times parameter. If one of the values for the two parameters is reached, the container path is no longer updated. For Linux Logtail V1.0.32 and later or Windows Logtail V1.0.32.0 and later, the default value is 3. Unit: seconds For Linux Logtail whose version is earlier than 1.0.32 or Windows Logtail whose version is earlier than 1.0.32.0, the default value is 10. Unit: seconds.	"docker_config_update_interval": 3
   max_docker_config_update_times	int	The maximum number of times that the container path can be updated within 3 minutes. By default, if the container path is updated more than three times within a 3-minute period, the container path cannot be updated again until 3 minutes later. For Linux Logtail V1.0.32 and later or Windows Logtail V1.0.32.0 and later, the default value is 10. For Linux Logtail whose version is earlier than 1.0.32 or Windows Logtail whose version is earlier than 1.0.32.0, the default value is 3.	"max_docker_config_update_times": 10
   DOCKER_HOST	String	The socket address that is used to communicate with Docker. You must configure the socket address by using an environment variable. This parameter is empty by default, which indicates that the default socket address unix:///var/run/docker.sock is used.	DOCKER_HOST=unix:///var/run/docker.sock
   CONTAINERD_SOCK_PATH	String	The socket address that is used to communicate with containerd. You must configure the socket address by using an environment variable. This parameter is empty by default, which indicates that the default socket address unix:///run/containerd/containerd.sock is used. If a K3s cluster is used, you can change the default socket address to the value provided in the example.	CONTAINERD_SOCK_PATH=/run/k3s/containerd/containerd.sock
   logreader_max_rotate_queue_size	Int	The maximum length of the queue in which a file is rotated. Default value: 20. If log collection is blocked or delayed, the files from which you want to collect logs are assigned the file handles and wait in the queue. If log collection is delayed and you need to manage the maximum disk usage, you can set this parameter to a smaller value. Warning If the number of delayed files exceeds the value of this parameter, Logtail does not collect logs from new files.	"logreader_max_rotate_queue_size" : 10
   force_release_deleted_file_fd_timeout	Int	The timeout period for the release of a file handle. If you want to release a file handle after a container exits or a file is deleted, you can configure this parameter. Unit: seconds. Default value: -1, which indicates that the feature is disabled. The value 0 indicates that file handles are immediately released. Unit: seconds. If you want to manage the maximum destruction latency of containerd containers, you can configure this parameter. Warning If log collection is delayed and the latency exceeds the specified threshold, data that is not collected is lost.	"force_release_deleted_file_fd_timeout" : 0
   default_max_inotify_watch_num	Int	The maximum number of directories that are monitored by using inotify. The directories include subdirectories. Default value: 3000.	"default_max_inotify_watch_num" : 5000
   data_endpoint_policy	string	The endpoint switchover policy for Logtail. Valid values: Note You can check whether the default endpoint is specified by the data_server_list parameter in the ilogtail_config.json file. For more information, see Startup configuration file (ilogtail_config.json). designated_first (default) If the default endpoint for a region is specified and the default endpoint is available, the system uses the default endpoint. If the default endpoint for a region is specified and the default endpoint is unavailable, the system automatically selects another available endpoint. If no default endpoint for a region is specified, the system automatically selects an available endpoint. designated_locked If the default endpoint for a region is specified, the system uses the default endpoint regardless of whether the endpoint is available. If no default endpoint for a region is specified, the system automatically selects an available endpoint. This parameter is available for only Linux Logtail V1.5.0 and later or Windows Logtail V1.5.0.0 and later.	"data_endpoint_policy" : "designated_first"
   inotify_black_list	Array<String>	The blacklist of inotify. Only exact match is supported. inotify does not listen to the directories specified in the blacklist.	"inotify_black_list": ["/tmp"]
   host_path_blacklist	String	The global host path blacklist. Only substring match is supported. On a Linux server, separate multiple substrings with colons (:). On a Windows server, separate multiple substrings with semicolons (;). For example, "host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-" indicates that NAS mounting-related data is not collected. This parameter is available for only Linux Logtail V1.8.0 and later or Windows Logtail V1.8.0.0 and later.	"host_path_blacklist" : "/volumes/kubernetes.io~csi/nas-"
   LOGTAIL_LOG_LEVEL	String	The level of logs. You must configure the log level by using an environment variable. This parameter is empty by default, which indicates that the log level is info. Valid values: trace, debug, info, warning, error, and fatal. This parameter is available for only Linux Logtail V1.8.0 and later or Windows Logtail V1.8.0.0 and later.	LOGTAIL_LOG_LEVEL=info

3. Restart Logtail for the new settings to take effect.

   /etc/init.d/ilogtaild stop && /etc/init.d/ilogtaild start                        

   After you restart Logtail, you can run the /etc/init.d/ilogtaild status command to check the status of Logtail.

Appendix: Environment variables

The following table describes the mappings between environment variables and the startup parameters of Logtail.