This topic describes how to troubleshoot the error that a Logtail machine group has no heartbeats in a host environment.
Examples of machine group installation
If an Elastic Compute Service (ECS) instance and your Simple Log Service project belong to the same Alibaba Cloud account and reside in the same region, create a machine group by following the instructions in Install Logtail and create a machine group.
If an ECS instance and your Simple Log Service project belong to different Alibaba Cloud accounts, an ECS instance and your Simple Log Service project belong to the same Alibaba Cloud account but reside in different regions, and in scenarios involving other clouds and self-managed servers, create a machine group by following the instructions in Create a machine group.
Troubleshooting preview
Step 1: Check whether Logtail runs as expected: Check whether Logtail runs as expected on the server.
Step 2: Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail: Check whether the IP address that is specified in the machine group is the same as the IP address in the app_info.json file of Logtail. If the IP addresses are different, a heartbeat failure occurs.
Step 3: Check whether the Logtail startup parameters are valid: Check whether the region that is specified in the
ilogtail_config.json
file is supported.Step 4: Check whether network connectivity is available: Check whether the server can connect to the address of the project.
Step 5: Check whether the system time of the server on which Logtail is installed is correct: If the system time is earlier or later than the local time, you must change the system time.
Step 6: Check whether a user identifier is configured: If an ECS instance and your Simple Log Service project belong to different Alibaba Cloud accounts, an ECS instance and your Simple Log Service project belong to the same Alibaba Cloud account but reside in different regions, and in scenarios involving other clouds and self-managed servers, you must check whether a user identifier is configured.
Step 7: Check whether a custom identifier is configured if you use a custom identifier-based machine group: If you use a custom identifier-based machine group, check whether a custom identifier is configured.
Step 8: Restart Logtail: After you perform the preceding operations, you must restart Logtail.
What to do next
If no logs are collected after the heartbeat status changes to OK, you can troubleshoot the issue based on the Logtail errors. For more information, see How do I view Logtail collection errors?
Step 1: Check whether Logtail runs as expected
Linux
Log on to the server on which Logtail is installed.
Run the following command:
ps -ef | grep ilogtail
If the following two records are returned, Logtail runs as expected. The records indicate the Logtail daemon process and the Logtail worker process.
UID PID PPID C STIME TTY TIME CMD ... root 12 1 0 Nov10 ? 00:00:00 /usr/local/ilogtail/ilogtail root 14 12 0 Nov10 ? 03:07:43 /usr/local/ilogtail/ilogtail ...
ImportantIf three or more records are returned, multiple Logtail processes are running on the current server. In this case, duplicate data may be collected. You must check whether the running Logtail processes meet your business requirements.
If the command output indicates that no Logtail processes are running, the issue may be due to the following reasons:
Logtail is installed but not started. For more information, see Start and stop Logtail.
Logtail is not installed. In this case, install Logtail. For more information, see Install Logtail on a Linux server.
ImportantWhen you install Logtail on a server, make sure that the operating system of the server is supported by Logtail. You can configure installation parameters based on the region of your Simple Log Service project and select an installation method based on the network type that you use. For more information about network types, see Select a network type.
Windows
Log on to the server on which Logtail is installed.
Open the Run window and enter
services.msc
to open the Services window.View the status of the LogtailDaemon or LogtailWorker service. For Logtail V1.0.0.0 or later, view the status of the LogtailDaemon service. For Logtail V0.x.x.x, view the status of the LogtailWorker service.
If the service is not in the Running state, the issue may be due to the following reasons:
Logtail is installed but not started. For more information, see Start and stop Logtail.
Logtail is not installed. In this case, install Logtail. For more information, see Install Logtail on a Windows server.
ImportantWhen you install Logtail on a server, make sure that the operating system of the server is supported by Logtail. You can configure installation parameters based on the region of your Simple Log Service project and select an installation method based on the network type that you use. For more information about network types, see Select a network type.
Step 2: Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail
Logtail obtains a Linux server IP address by using one of the following methods:
If the hostname of the server is not mapped to an IP address, Logtail obtains the IP address of the first network interface controller (NIC) of the server.
If you want to specify a custom IP address, you can configure the
working_ip
field in theilogtail_config.json
file obtained in Step 3. After you configure the working_ip field, the value of theip
field in theapp_info.json
file is automatically updated with the value of theworking_ip
field. For more information about the working_ip field, see Configure the startup parameters of Logtail.If the hostname of the server is mapped to an IP address, Logtail obtains the IP address from the
/etc/hosts
file of the server. The /etc/hosts file includes hostname-IP address mapping settings.
Obtain the value of the
ip
field in the app_info.json file.The following table describes the default paths to the app_info.json file in different operating systems.
Operating system
Logtail
Path to the app_info.json file
Linux
Logtail for 64-bit Linux
/usr/local/ilogtail/app_info.json
64-bit Windows
Logtail for 64-bit Windows
C:\Program Files\Alibaba\Logtail\app_info.json
Logtail for 32-bit Windows
C:\Program Files (x86)\Alibaba\Logtail\app_info.json
32-bit Windows
Logtail for 32-bit Windows
C:\Program Files\Alibaba\Logtail\app_info.json
The value of the
ip
field specifies the IP address that is obtained by Logtail.{ "UUID" : "", "hostname" : "iZ8vbdlzf******azuhZ", "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11", "ip" : "172.**.**.200", "logtail_version" : "1.3.1", "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64", "update_time" : "2022-12-27 05:38:33" }
Check whether the IP address that is specified in the machine group is the same as the IP address that is obtained by Logtail.
Simple Log Service machine groups are classified into IP address-based machine groups and custom identifier-based machine groups. For more information, see Introduction to machine groups.
IP address-based machine group: Check whether the IP address obtained in the previous step is included in the value of the IP Address field.
If the value of the IP Address field is another IP address of Logtail such as a public IP address, change the value to the IP address that is obtained by Logtail. Then, check whether the heartbeat of the machine group can be detected. If the heartbeat can be detected, you can stop troubleshooting.
Custom identifier-based machine group: Check whether the IP address obtained in the previous step is included in the Machine Group Status section. If the heartbeat status is OK, you can stop troubleshooting.
Step 3: Check whether the Logtail startup parameters are valid
The ilogtail_config.json file records the startup parameters of Logtail.
Log on to the server on which Logtail is installed.
Find the ilogtail_config.json file.
The following table describes the default paths to the ilogtail_config.json file in different operating systems.
Operating system
Logtail
Path to the ilogtail_config.json file
Linux
Logtail for 64-bit Linux
/usr/local/ilogtail/ilogtail_config.json
64-bit Windows
Logtail for 64-bit Windows
\ilogtail_config.json
Logtail for 32-bit Windows
C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json
32-bit Windows
Logtail for 32-bit Windows
C:\Program Files\Alibaba\Logtail\ilogtail_config.json
Open the ilogtail_config.json file and check whether the parameters in the configuration file are valid.
{ "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com", "data_server_list" : [ { "cluster" : "<Region to which the project belongs>", "endpoint" : "<endpoint>" } ], ... }
If the configurations of the startup parameters in the ilogtail_config.json file comply with the descriptions in the following table, the startup parameters of Logtail are valid.
If the startup parameters of Logtail are invalid, modify the ilogtail_config.json file based on the descriptions in the following table and restart Logtail. For more information, see Restart Logtail.
For more information about the supported regions, see Supported regions.
Scenario
Network type
<config_region>
<endpoint>
The server is an ECS instance that resides in the same region as the project.
Alibaba Cloud internal network
<Region to which the project belongs>-intranet
<Region to which the project belongs>-intranet.log.aliyuncs.com
Other scenarios
Internet
<Region to which the project belongs>
<Region to which the project belongs>.log.aliyuncs.com
Transfer acceleration
log-global.aliyuncs.com
Step 4: Check whether network connectivity is available
Logtail can upload data to Simple Log Service only if the server on which Logtail is installed can connect to the following addresses.
If you use an internal network, you must append -intranet
to <endpoint>
.
The address that is specified by the
config_server_address
field in theilogtail_config.json
file. Only the HTTPS version specified by the field value is supported.The address in the
http://<Project name>.<endpoint>
format.You can view the name and the region of the project by following the instructions in the figure.
<endpoint>
is the address that is specified by thedata_server_list.endpoint
field in theilogtail_config.json
file.
The address in the
http://ali-<Region to which the project belongs>-sls-admin.<endpoint>
format.<endpoint>
is the address that is specified by thedata_server_list.endpoint
field in theilogtail_config.json
file.
To check whether network connectivity is available, perform the following operations:
Linux
Log on to the server on which Logtail is installed.
Run the
curl
command to connect to the preceding addresses in sequence:curl http://<Project name>.cn-hangzhou-intranet.log.aliyuncs.com
If each command output contains information that is similar to the following code, network connectivity is available.
{"Error":{"Code":"OLSInvalidMethod","Message":"The script name is invalid : /","RequestId":"5D****09"}}
If network connectivity is unavailable, check whether ports 80 and 443 in the network environment are enabled, whether the destination address is blocked, and whether other network-related configurations such as Domain Name System (DNS) configurations and security groups are valid.
Windows
Log on to the server on which Logtail is installed.
Run the
telnet
command to connect to the preceding addresses in sequence:telnet <Project name>.cn-hangzhou-intranet.log.aliyuncs.com 80 # If HTTPS is used, specify port 443.
If each command output contains information that is similar to the following code, network connectivity is available.
Trying 100*0*7*5... Connected to xxx. Escape character is '^]'.
If network connectivity is unavailable, check whether ports 80 and 443 in the network environment are enabled, whether the destination address is blocked, and whether other network-related configurations such as DNS configurations and security groups are valid.
Step 5: Check whether the system time of the server on which Logtail is installed is correct
Linux
Log on to the server on which Logtail is installed.
Run the
date
command to view the system time:Wed Dec 28 06:59:26 UTC 2022
If the system time is earlier or later than the local time, perform the following operations:
Change the system time to the local time.
If you cannot change the system time, add
"enable_log_time_auto_adjust": true
to the ilogtail_config.json file. After you modify the file, restart Logtail. For more information, see Restart Logtail. For more information about the path to the ilogtail_config.json file, see Step 3: Check whether the Logtail startup parameters are valid.
Windows
Log on to the server on which Logtail is installed.
View the time information in the taskbar in the lower-right corner of the desktop.
Change the system time to the local time.
If you cannot change the system time, add "enable_log_time_auto_adjust": true to the ilogtail_config.json file. After you modify the file, restart Logtail. For more information, see Restart Logtail. For more information about the path to the ilogtail_config.json file, see Step 3: Check whether the Logtail startup parameters are valid.
Step 6: Check whether a user identifier is configured
If an ECS instance and your Simple Log Service project belong to different Alibaba Cloud accounts, an ECS instance and your Simple Log Service project belong to the same Alibaba Cloud account but reside in different regions, and in scenarios involving other clouds and self-managed servers, you must check whether a user identifier is configured.
A user identifier must be the ID of an Alibaba Cloud account. For more information, see Configure a user identifier.
You can use a user identifier file in a specified directory to check whether a user identifier is configured. If the output is empty, you must check whether a user identifier file exists in a specified directory.
The path to a user identifier file varies based on the operating system.
Linux: /etc/ilogtail/users/
Windows: C:\LogtailData\users\
If no user identifier file exists in the specified directory or a user identifier file is invalid, perform the following operations to fix the issue:
Linux: Run the
cd /etc/ilogtail/users/ && touch <uid>
command to create a user identifier file.<uid>
specifies the ID of the Alibaba Cloud account to which the project belongs.Windows: Go to the
C:\LogtailData\users\
directory and create an empty file named<uid>
.<uid>
specifies the ID of the Alibaba Cloud account to which the project belongs.
If the specified directory contains a file named based on the ID of the Alibaba Cloud account to which the project belongs, the user identifier is configured and valid.
Step 7: Check whether a custom identifier is configured if you use a custom identifier-based machine group
You can use the user_defined_id file in a specified directory to check whether a custom identifier is configured for your server.
If the output is empty, you must check whether the
user_defined_id
file exists or whether a custom identifier is configured in the file.NoteThe path to the user_defined_id file varies based on the operating system.
Linux: /etc/ilogtail/user_defined_id
Windows: C:\LogtailData\user_defined_id
If the user_defined_id file does not exist, create a file named user_defined_id and add the custom identifier of the machine group to the file. For more information, see Configure a custom identifier for your servers.
If no custom identifier exists in the user_defined_id file or the custom identifier is invalid, add a line to the file and enter the custom identifier of the machine group. For more information, see Configure a custom identifier for your servers.
If the user_defined_id file contains the custom identifier that you configured for the machine group, the custom identifier is configured and valid.
Step 8: Restart Logtail
After you perform the preceding operations, you must restart Logtail.
Linux
Log on to the server on which Logtail is installed.
Run the following command:
sudo /etc/init.d/ilogtaild restart
Windows
Log on to the server on which Logtail is installed.
Open the Run window and enter
services.msc
to open the Services window.Restart the LogtailDaemon or LogtailWorker service. For Logtail V1.0.0.0 or later, restart the LogtailDaemon service. For Logtail V0.x.x.x, restart the LogtailWorker service.
What to do next
If the error persists after troubleshooting, submit a ticket.