Simple Log Service:Troubleshoot Logtail machine group issues on a host

Machine group installation examples

Troubleshooting guide

1. Step 1: Check whether Logtail runs as expected. Check whether Logtail is running as expected on the server.

2. Step 2: Confirm that the IP address in the machine group is the one retrieved by Logtail. Confirm that the IP address in the machine group is the same as the IP address in the app_info.json file of Logtail. A mismatch can cause heartbeat failure.

3. Step 3: Check whether the Logtail startup parameters are correct. Check whether the project region configured in the ilogtail_config.json file is correct.

4. Step 4: Check the network connection. Check whether the network connection between the server and the project is working correctly.

5. Step 5: Check whether the system time of the Logtail server is correct. If the system time is significantly faster or slower than the current time, you must change it.

6. Step 6: Check the user identifier for cross-account collection. If the server is not an ECS instance, or if the ECS instance and the project belong to different Alibaba Cloud accounts, you must check whether a correct user identifier is configured.

7. Step 7: If you use a custom identifier-based machine group, check whether a custom identifier is configured. If the machine group is a custom identifier-based machine group, check whether a custom identifier is configured.

8. Step 8: Restart Logtail. After you complete the preceding modifications, you must restart Logtail.

What to do next

If logs are still not collected after the heartbeat status is OK, you can troubleshoot the issue by viewing Logtail error messages. For more information, see View Logtail collection errors.

Step 1: Check whether Logtail runs as expected

1. Retrieve the value of the ip field from the app_info.json file.

   The following table describes the default paths of this file for different operating systems.

   Operating system	Logtail	Path to the app_info.json file
   Linux	Logtail (64-bit program)	/usr/local/ilogtail/app_info.json
   Windows (64-bit OS)	Logtail (64-bit program)	C:\Program Files\Alibaba\Logtail\app_info.json
   	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json
   Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\app_info.json

   Logtail records the retrieved IP address in the ip field of the app_info.json file.

   {
     "UUID" : "",
     "hostname" : "iZ8vbdlzf******azuhZ",
     "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11",
     "ip" : "172.**.**.200",
     "logtail_version" : "1.3.1",
     "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64",
     "update_time" : "2022-12-27 05:38:33"
   }

2. Confirm that the machine group uses the IP address retrieved by Logtail.

   Simple Log Service machine groups include IP-based machine groups and custom identifier-based machine groups. For more information, see Machine groups.

   • IP-based machine group: Verify that the IP Address field contains the IP address from the previous step.

     If the IP address is not included, confirm the correct IP address for the host. If the IP Address text box contains another IP address for the target Logtail, such as a public IP address, modify the IP address in the IP-based machine group. If the IP address from the previous step is incorrect, change the working_ip parameter as described in Set Logtail startup parameters and restart Logtail. Then, check if the machine heartbeat is normal. If the heartbeat is normal, the troubleshooting is complete.

   • Custom identifier-based machine group: In the Machine Group Status section, verify that the IP address obtained in the previous step is listed. If the Heartbeat status is OK, you can end the troubleshooting process.

The ilogtail_config.json file records the startup parameters for Logtail.

1. Log on to the server where Logtail is installed.

2. Find the ilogtail_config.json file.

   The following table describes the default paths of this file for different operating systems.

   Operating system	Logtail	Path to the ilogtail_config.json file
   Linux	Logtail (64-bit program)	/usr/local/ilogtail/ilogtail_config.json
   Windows (64-bit OS)	Logtail (64-bit program)	\ilogtail_config.json
   	Logtail (32-bit program)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json
   Windows (32-bit OS)	Logtail (32-bit program)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

   1. Open the ilogtail_config.json file and confirm that the configuration parameters are correct.

      {
        "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com",
        "data_server_list" :
        [
          {
            "cluster" : "<project_region>",
            "endpoint" : "<endpoint>"
          }
        ],
        ...
      }

      • If the startup parameters in the ilogtail_config.json file match the descriptions in the following table, the Logtail startup parameters are correct.

      • If the Logtail startup parameters are incorrect, modify the ilogtail_config.json file based on the following table, and then restart Logtail. For more information, see Restart Logtail.

        For more information about project regions, see Regions and endpoints.

        Scenario	Network type	<config_region>	<endpoint>
        The server is an ECS instance and is in the same region as the project.	Alibaba Cloud private network	<project_region>-intranet	<project_region>-intranet.log.aliyuncs.com
        Other scenarios	Public network	<project_region>	<project_region>.log.aliyuncs.com
        	Transfer acceleration		log-global.aliyuncs.com

To upload data using Logtail, ensure that the server where Logtail is installed can connect to the following addresses.

1. The address specified by the config_server_address field in the ilogtail_config.json file, and its HTTPS version.

2. http://<project_name>.<endpoint>.

   • The following describes how to view the Project name and region.

     

   • <endpoint> is the address specified by the data_server_list.endpoint field in the ilogtail_config.json file.

3. http://ali-<project_region>-sls-admin.<endpoint>. <endpoint> is the address specified by the data_server_list.endpoint field in the ilogtail_config.json file.

Check the network connection and resolve issues as follows:

Linux

1. Log on to the server where Logtail is installed.

2. Run the date command to view the system time.

   Wed Dec 28 06:59:26 UTC 2022

   If the system time is significantly faster or slower than the current time, you can try the following solutions:

   • Adjust the system time to the current time.

   • If you cannot change the system time, add the "enable_log_time_auto_adjust": true configuration item to the ilogtail_config.json file. After the modification, restart Logtail. For more information, see Restart Logtail. For information about the path to the ilogtail_config.json file, see Step 3: Check whether the Logtail startup parameters are correct.

Windows

1. Log on to the server where Logtail is installed.

2. View the time information in the taskbar at the bottom right of the desktop.

   • Adjust the system time to the current time.

   • If you cannot change the system time, add the "enable_log_time_auto_adjust": true configuration item to the ilogtail_config.json file. After the modification, restart Logtail. For more information, see Restart Logtail. For information about the path to the ilogtail_config.json file, see Step 3: Check whether the Logtail startup parameters are correct.

Step 6: Check the user identifier for cross-account collection

Check the user identifier file in the specified directory to determine whether a user identifier exists. If the directory is empty, check whether a user identifier file is present in the specified path. The user identifier grants the account permission to access this server.

• If no user identifier file exists in the specified path or the file is configured incorrectly, you can resolve the issue as follows:

  • For Linux: Run the cd /etc/ilogtail/users/ && touch <uid> command to create the user identifier file. <uid> is the Alibaba Cloud account ID of the project owner.

  • For Windows: Go to the C:\LogtailData\users\ directory and create an empty file named <uid>. <uid> is the Alibaba Cloud account ID of the project owner.

• If a file named with the Alibaba Cloud account ID of the project owner exists in the specified path, the user identifier is configured correctly.

Step 7: If you use a custom identifier-based machine group, check whether a custom identifier is configured

If you use a custom identifier-based machine group, you can check the user_defined_id file in the specified directory to determine whether a custom identifier is configured on the server.

• If the file is empty, check whether the user_defined_id file exists or whether a custom identifier is configured in the file.

  Note

  The paths to the user_defined_id file are as follows:

  • Linux: /etc/ilogtail/user_defined_id

  • Windows: C:\LogtailData\user_defined_id

  • If the user_defined_id file does not exist, create a user_defined_id file and enter the custom identifier of the machine group in the file. For more information, see Configure a custom identifier.

  • If the user_defined_id file has no custom identifier or the identifier is incorrect, add a new line to the file and enter the custom identifier of the machine group. For more information, see Configure a custom identifier.

  • If the user_defined_id file already contains the custom identifier that you set in the machine group, the custom identifier is configured correctly.

Step 8: Restart Logtail

After you complete the preceding modifications, you must restart Logtail.