Logtail configuration files and record files - Simple Log Service

This topic describes the basic configuration files and record files of Logtail. When Logtail is active, it uses the configuration files and generates record files.

Startup configuration file (ilogtail_config.json)

The ilogtail_config.json file is used to configure the startup parameters of Logtail. The file is in the JSON format. For more information, see Configure the startup parameters of Logtail.

Important

The file must be a valid JSON file. Otherwise, Logtail cannot be started.
If you modify the file, you must restart Logtail for the modification to take effect. For more information, see Restart Logtail.
By default, Logtail uses the HTTP protocol to communicate with the servers on control planes and data planes, and uses the HTTPS protocol to perform authentication with the servers.
- To ensure security, you can use the HTTPS protocol to communicate with the servers. In this case, you must set the config_server_address and data_server_list.endpoint explicit parameters to https.
- If you use HTTPS to transmit data, the transmission latency increases. We recommend that you use HTTPS only in the required scenarios.

After Logtail is installed on a server, you can perform the following operations in the ilogtail_config.json file:

Modify the runtime parameters of Logtail.
Check whether the installation commands meet your requirements.
The values of the config_server_address and data_server_list parameters in the ilogtail_config.json file vary based on the installation command that you select. If the region in the installation command is different from the region where your Simple Log Service project resides or the address in the command cannot be accessed, the installation command that you select does not meet your requirements. In this case, Logtail cannot collect logs and must be reinstalled.

File path

Host environment

Operating system	Logtail	Path to the ilogtail_config.json file
Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail_config.json
64-bit Windows	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json
64-bit Windows	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json Note You can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
32-bit Windows	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

Container environment
The ilogtail_config.json file is stored in a Logtail container. The file path is specified in the ALIYUN_LOGTAIL_CONFIG environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_CONFIG command to view the file path. Example: /etc/ilogtail/conf/cn-hangzhou/ilogtail_config.json.

Sample file

$cat /usr/local/ilogtail/ilogtail_config.json
{
    "config_server_address" : "http://logtail.cn-hangzhou-intranet.log.aliyuncs.com",
    "data_server_list" :
    [
        {
            "cluster" : "cn-hangzhou",
            "endpoint" : "cn-hangzhou-intranet.log.aliyuncs.com"
        }
    ],
    "cpu_usage_limit" : 0.4,
    "mem_usage_limit" : 100,
    "max_bytes_per_sec" : 2097152,
    "process_thread_count" : 1,
    "send_request_concurrency" : 4,
    "streamlog_open" : false
}

User identifier file

The user identifier file contains the ID of your Alibaba Cloud account. The file specifies that the account is authorized to access the server on which Logtail is installed and collect logs from the server. For more information, see Configure a user identifier.

Important

If you want to collect logs from an Elastic Compute Service (ECS) instance that belongs to another Alibaba Cloud account, a server that is deployed in a self-managed data center, or a server that is provided by a third-party cloud service provider, you must configure a user identifier for your server.
You must specify the ID of an Alibaba Cloud account as a user identifier in the user identifier file. You cannot specify the ID of a Resource Access Management (RAM) user as a user identifier.
You must specify the name of the user identifier file. You do not need to specify the file extension.
You can specify multiple user identifiers for a server. However, you can specify only one user identifier for a Logtail container.

File path
- Host environment
  - Linux: /etc/ilogtail/users/
  - Windows: C:\LogtailData\users\
- Container environment
  The file is stored in a Logtail container. The file path is specified in the ALIYUN_LOGTAIL_USER_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_ID command to view the file path.
Sample file
```
$ls /etc/ilogtail/users/
```

Custom identifier file (user_defined_id)

The user_defined_id file is used to configure a custom identifier. For more information, see Create a custom identifier-based machine group.

Important

When you create a custom identifier-based machine group, you must configure the user_defined_id file.

File path
- Host environment
  - Linux: /etc/ilogtail/user_defined_id
  - Windows: C:\LogtailData\user_defined_id
- Container environment
  The file is stored in a Logtail container. The file path is specified in the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_DEFINED_ID command to view the file path.

Sample file

$cat /etc/ilogtail/user_defined_id
aliyun-ecs-rs1e16355

Logtail configuration file (user_log_config.json)

The user_log_config.json file records the information about a Logtail configuration that is received by Logtail from Simple Log Service. The file is in the JSON format and is updated together with configuration updates. You can use the user_log_config.json file to check whether the Logtail configuration is delivered to the server on which Logtail is installed. If the Logtail configuration file exists and the configurations in the file are the same as the settings of the Logtail configuration in Simple Log Service, the Logtail configuration is delivered.

Important

We recommend that you do not modify the Logtail configuration file unless you need to specify sensitive information, such as your AccessKey pair and database password.

File path

Host environment

Operating system	Logtail	Path to the user_log_config.json file
Linux	Logtail (64-bit)	/usr/local/ilogtail/user_log_config.json
64-bit Windows	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\user_log_config.json
64-bit Windows	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\user_log_config.json Note You can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
32-bit Windows	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\user_log_config.json

Container environment
The user_log_config.json file is stored in a Logtail container. The file path is /usr/local/ilogtail/user_log_config.json.

Sample file

$cat /usr/local/ilogtail/user_log_config.json
{
   "metrics" : {
      "##1.0##k8s-log-c12ba2028*****939f0b$app-java" : {
         "aliuid" : "16542189*****50",
         "category" : "app-java",
         "create_time" : 1534739165,
         "defaultEndpoint" : "cn-hangzhou-intranet.log.aliyuncs.com",
         "delay_alarm_bytes" : 0,
         "enable" : true,
         "enable_tag" : true,
         "filter_keys" : [],
         "filter_regs" : [],
         "group_topic" : "",
         "local_storage" : true,
         "log_type" : "plugin",
         "log_tz" : "",
         "max_send_rate" : -1,
         "merge_type" : "topic",
         "plugin" : {
            "inputs" : [
               {
                  "detail" : {
                     "IncludeEnv" : {
                        "aliyun_logs_app-java" : "stdout"
                     },
                     "IncludeLable" : {
                        "io.kubernetes.container.name" : "java-log-demo-2",
                        "io.kubernetes.pod.namespace" : "default"
                     },
                     "Stderr" : true,
                     "Stdout" : true
                  },
                  "type" : "service_docker_stdout"
               }
            ]
         },
         "priority" : 0,
         "project_name" : "k8s-log-c12ba2028c*****ac1286939f0b",
         "raw_log" : false,
         "region" : "cn-hangzhou",
         "send_rate_expire" : 0,
         "sensitive_keys" : [],
         "tz_adjust" : false,
         "version" : 1
      }
   }
}

AppInfo record file (app_info.json)

The app_info.json file records the information about Logtail, such as the startup time, obtained IP address, and hostname.

If you map the IP address of a server to a hostname in the/etc/hosts file of the server, Logtail directly obtains the IP address. If you do not map the IP address of a server to the hostname, Logtail obtains the IP address of the first network interface controller (NIC) on the server.

Important

The AppInfo record file records only the basic information about Logtail. If you modify the file, the IP address that is obtained by Logtail does not change.
If you modify the network settings of a server, such as the hostname, you must restart Logtail to obtain a new IP address.

File path

Host environment

Operating system	Logtail	Path to the app_info.json file
Linux	Logtail (64-bit)	/usr/local/ilogtail/app_info.json
64-bit Windows	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\app_info.json
64-bit Windows	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json Note You can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
32-bit Windows	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\app_info.json

Container environment
The app_info.json file is stored in a Logtail container. The file path is /usr/local/ilogtail/app_info.json.

Sample file

$cat /usr/local/ilogtail/app_info.json
{
   "UUID" : "",
   "hostname" : "logtail-ds-slpn8",
   "instance_id" : "E5F93BC6-B024-11E8-8831-0A58AC14039E_1**.***.***.***_1536053315",
   "ip" : "1**.***.***.***",
   "logtail_version" : "0.16.13",
   "os" : "Linux; 3.10.0-693.2.2.el7.x86_64; #1 SMP Tue Sep 12 22:26:13 UTC 2017; x86_64",
   "update_time" : "2018-09-04 09:28:36"
}

Field	Description
UUID	The serial number of the server.
hostname	The hostname.
instance_id	The unique identifier of Logtail. The identifier is randomly generated.
ip	The IP address that is obtained by Logtail. If Logtail does not obtain an IP address, this field is empty, and Logtail cannot run as expected. In this case, you must specify an IP address for the server and restart Logtail. Note If you create an IP address-based machine group, make sure that the IP address that you specify for the machine group is consistent with the value of this field. If the IP address that you specify for the machine group is inconsistent with the value of this field, log on to the Simple Log Service console and modify the IP address of the machine group. Wait for 1 minute and view the IP address of the machine group in the Simple Log Service console.
logtail_version	The version of Logtail.
os	The version of the operating system.
update_time	The most recent startup time of Logtail.

Logtail operational log file (ilogtail.LOG)

The ilogtail.LOG file records the operational logs of Logtail. The levels of logs in ascending order are INFO, WARN, and ERROR. You can ignore logs at the INFO level.

If an error occurs during log collection, troubleshoot the error based on the error type and Logtail operational logs. For more information, see How do I view Logtail collection errors?

Note

You must upload the ilogtail.LOG file when you submit a ticket to troubleshoot a Logtail collection error.

File path

Host environment

Operating system	Logtail	Path to the ilogtail.LOG file
Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail.LOG
64-bit Windows	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\ilogtail.LOG
64-bit Windows	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail.LOG Note You can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
32-bit Windows	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail.LOG

Container environment
The ilogtail.LOG file is stored in a Logtail container. The file path is /usr/local/ilogtail/ilogtail.LOG.

Sample file

$tail /usr/local/ilogtail/ilogtail.LOG
[2018-09-13 01:13:59.024679]    [INFO]    [3155]    [build/release64/sls/ilogtail/elogtail.cpp:123]    change working dir:/usr/local/ilogtail/
[2018-09-13 01:13:59.025443]    [INFO]    [3155]    [build/release64/sls/ilogtail/AppConfig.cpp:175]    load logtail config file, path:/etc/ilogtail/conf/ap-southeast-1/ilogtail_config.json
[2018-09-13 01:13:59.025460]    [INFO]    [3155]    [build/release64/sls/ilogtail/AppConfig.cpp:176]    load logtail config file, detail:{
   "config_server_address" : "http://logtail.ap-southeast-1-intranet.log.aliyuncs.com",
   "data_server_list" : [
      {
         "cluster" : "ap-southeast-1",
         "endpoint" : "ap-southeast-1-intranet.log.aliyuncs.com"
      }
]

Operational log file of Logtail plug-ins (logtail_plugin.LOG)

The logtail_plugin.LOG file records the operational logs of Logtail plug-ins. The levels of logs in ascending order are INFO, WARN, and ERROR. You can ignore logs at the INFO level.

If the CANAL_RUNTIME_ALARM error message appears when you diagnose log collection errors, you can troubleshoot the error based on the logtail_plugin.LOG file.

Note

You must upload the file when you submit a ticket to troubleshoot a Logtail plug-in error.

File path

Host environment

Operating system	Logtail	Path to the logtail_plugin.LOG file
Linux	Logtail (64-bit)	/usr/local/ilogtail/logtail_plugin.LOG
64-bit Windows	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\logtail_plugin.LOG
64-bit Windows	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\logtail_plugin.LOG Note You can run 32-bit and 64-bit applications in 64-bit Windows. To ensure compatibility, the operating system stores 32-bit applications in a separate x86 directory.
32-bit Windows	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\logtail_plugin.LOG

Container environment
The logtail_plugin.LOG file is stored in a Logtail container. The file path is /usr/local/ilogtail/logtail_plugin.LOG.

Sample file

$tail /usr/local/ilogtail/logtail_plugin.LOG
2018-09-13 02:55:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
2018-09-13 02:55:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop
2018-09-13 03:00:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
2018-09-13 03:00:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop
2018-09-13 03:03:26 [INF] [log_file_reader.go:221] [ReadOpen] [##1.0##sls-zc-test-hz-pub$docker-stdout-config,k8s-stdout]    open file for read, file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379573    status:794354-64769-40379963
2018-09-13 03:03:26 [INF] [log_file_reader.go:221] [ReadOpen] [##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$docker-stdout-config,k8s-stdout]    open file for read, file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379573    status:794354-64769-40379963
2018-09-13 03:04:26 [INF] [log_file_reader.go:308] [CloseFile] [##1.0##sls-zc-test-hz-pub$docker-stdout-config,k8s-stdout]    close file, reason:no read timeout    file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379963    status:794354-64769-40379963
2018-09-13 03:04:27 [INF] [log_file_reader.go:308] [CloseFile] [##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$docker-stdout-config,k8s-stdout]    close file, reason:no read timeout    file:/logtail_host/var/lib/docker/containers/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624/7f46afec6a14de39b59ee9cdfbfa8a70c2fa26f1148b2e2f31bd3410f5b2d624-json.log    offset:40379963    status:794354-64769-40379963
2018-09-13 03:05:30 [INF] [docker_center.go:525] [func1] docker fetch all:start
2018-09-13 03:05:30 [INF] [docker_center.go:529] [func1] docker fetch all:stop

Container path mapping file (docker_path_config.json)

The docker_path_config.json file is created only when you collect container logs. The file records the path mappings between container log files and host log files. The file is in the JSON format.

If the DOCKER_FILE_MAPPING_ALARM error message appears when you diagnose log collection errors, you can troubleshoot the error based on the docker_path_config.json file. The error message indicates that Docker files cannot be mapped to host files.

Note

The docker_path_config.json file is a record file. Modifications to this file do not take effect. If you delete this file, another file is automatically created without service interruptions.
You must upload the file when you submit a ticket to troubleshoot a container log collection error.

File path
/usr/local/ilogtail/docker_path_config.json

Sample file

$cat /usr/local/ilogtail/docker_path_config.json
{
   "detail" : [
      {
         "config_name" : "##1.0##k8s-log-c12ba2028cfb444238cd9ac1286939f0b$nginx",
         "container_id" : "df19c06e854a0725ea7fca7e0378b0450f7bd3122f94fe3e754d8483fd330d10",
         "params" : "{\n   \"ID\" : \"df19c06e854a0725ea7fca7e0378b0450f7bd3122f94fe3e754d8483fd330d10\",\n   \"Path\" : \"/logtail_host/var/lib/docker/overlay2/947db346695a1f65e63e582ecfd10ae1f57019a1b99260b6c83d00fcd1892874/diff/var/log\",\n   \"Tags\" : [\n      \"nginx-type\",\n      \"access-log\",\n      \"_image_name_\",\n      \"registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest\",\n      \"_container_name_\",\n      \"nginx-log-demo\",\n      \"_pod_name_\",\n      \"nginx-log-demo-h2lzc\",\n      \"_namespace_\",\n      \"default\",\n      \"_pod_uid_\",\n      \"87e56ac3-b65b-11e8-b172-00163f008685\",\n      \"_container_ip_\",\n      \"172.20.4.224\",\n      \"purpose\",\n      \"test\"\n   ]\n}\n"
      }
   ],
   "version" : "0.1.0"
}