How do I collect multiple copies of logs in a file? - Simple Log Service

By default, a file can match only one Logtail configuration for collection. This topic describes how to collect multiple copies of logs in a file.

Background information

If a file matches multiple Logtail configurations for collection, the usage of CPU, memory, disk I/O, and network I/O resources greatly increases. This affects the performance of other services that are deployed on the same server as Logtail. Therefore, a file can match only one Logtail configuration for collection in default scenarios.

Solutions

Store logs in different Logstores

You do not need to create multiple Logtail configurations. You can use the data transformation feature of Simple Log Service to replicate logs. This prevents the performance of other services on the same server from being affected. For more information, see Replicate data from a Logstore.

Create multiple Logtail configurations

Host scenarios

Create a symbolic link for a directory
Create a symbolic link for the directory in which the required file is stored. Then, you can specify the real path in one Logtail configuration and specify the symbolic link in the other Logtail configuration. For example, you can create the /home/log/nginx/link_log symbolic link for the /home/log/nginx/log directory.
```
ln -s /home/log/nginx/log /home/log/nginx/link_log
```
Add settings in Logtail configurations to forcefully collect logs
When you create a Logtail configuration, turn on Allow File to Be Collected for Multiple Times in the Input Configurations section. For more information, see Collect text logs from servers.

Container scenarios

For more information about how to collect container logs, see Use the Simple Log Service console to collect container text logs in DaemonSet mode and Use the Simple Log Service console to collect container text logs in Sidecar mode.

Console
When you create a Logtail configuration, turn on Allow File to Be Collected for Multiple Times in the Input Configurations section.
CRD
Create multiple custom resource definitions (CRDs).