Simple Log Service:How do I troubleshoot an error that is related to a Logtail machine group in a host environment?

Machine group installation examples

• Machine group configuration: ECS and Simple Log Service project are under the same account and region.

• Collect logs across regions using Logtail: ECS and Simple Log Service project are under the same account but in different regions.

• Collect logs across Alibaba Cloud accounts using Logtail: Projects and ECS servers are under different accounts, or involve other clouds or self-managed servers.

Troubleshooting preview

1. Step 1: Verify that Logtail is operational: Ensure Logtail is functioning correctly on the server.

2. Step 2: Confirm the IP address in the machine group matches the IP address retrieved by Logtail: Ensure the IP address listed in the machine group matches the IP address in Logtail's app_info.json file. Discrepancies will cause heartbeat failures.

3. Step 3: Verify the Validity of Logtail Startup Parameters: Ensure that the project region specified in the ilogtail_config.json file is correct.

4. Step 4: Confirm network connectivity: Check the network connection between the server and the project.

5. Step 5: Verify the server's system time is accurate: Adjust the system time if it deviates from the local time.

6. Step 6: Check for a user identifier: If the project and ECS server are under different accounts or involve other clouds or self-managed servers, verify the presence of a user identifier.

7. Step 7: Ensure a custom identifier is set up: If using a custom identifier-based machine group, confirm a custom identifier is in place.

Step 1: Check whether Logtail is running as expected

1. Retrieve the value of the ip field in the app_info.json file.

   The default file path varies by operating system, as shown in the table below:

   Operating system	Logtail	Path of the app_info.json file
   Linux	Logtail (64-bit)	/usr/local/ilogtail/app_info.json
   Windows (64-bit operating system)	Logtail (64-bit)	C:\Program Files\Alibaba\Logtail\app_info.json
   	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\app_info.json
   Windows (32-bit operating system)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\app_info.json

   Logtail records the retrieved IP address in the ip field of the app_info.json file.

   {
     "UUID" : "",
     "hostname" : "iZ8vbdlzf******azuhZ",
     "instance_id" : "E9633380-***********-00163E1AA597_172.16.2.200_166****11",
     "ip" : "172.**.**.200",
     "logtail_version" : "1.3.1",
     "os" : "Linux; 4.19.91-26.1.al7.x86_64; #1 SMP Tue Jul 26 17:52:28 CST 2022; x86_64",
     "update_time" : "2022-12-27 05:38:33"
   }

2. Verify that the IP address retrieved by Logtail is listed in the machine group.

   Simple Log Service offers two types of machine groups: those based on IP addresses and those using custom identifiers. For more information, see what is a machine group.

   • IP address-based machine group: Confirm that the IP Address field includes the IP address retrieved in the previous step.

     If the IP Address field lists different IP addresses, such as a public IP, update it to the IP address retrieved by Logtail and check for normal heartbeat activity. If the heartbeat is normal, troubleshooting can cease.

   • Custom identifier-based machine group: Ensure the Machine Group Status includes the IP address retrieved in the previous step. If the Heartbeat displays OK, the troubleshooting process can be concluded.

The ilogtail_config.json file contains Logtail's startup parameters.

1. Log on to the server where Logtail is installed.

2. Locate the ilogtail_config.json file.

   The default file path varies by operating system, as shown in the table below:

   Operating system	Logtail	Path of the ilogtail_config.json file
   Linux	Logtail (64-bit)	/usr/local/ilogtail/ilogtail_config.json
   Windows (64-bit operating system)	Logtail (64-bit)	\ilogtail_config.json
   	Logtail (32-bit)	C:\Program Files (x86)\Alibaba\Logtail\ilogtail_config.json
   Windows (32-bit operating system)	Logtail (32-bit)	C:\Program Files\Alibaba\Logtail\ilogtail_config.json

   1. Open the ilogtail_config.json file and check the validity of the parameters within.

      {
        "config_server_address" : "http://logtail.<config_region>.log.aliyuncs.com",
        "data_server_list" :
        [
          {
            "cluster" : "<project region>",
            "endpoint" : "<endpoint>"
          }
        ],
        ...
      }

      • If the startup parameter configurations in the ilogtail_config.json file align with the descriptions in the table below, the parameters are valid.

      • If the startup parameters are invalid, modify the ilogtail_config.json file according to the descriptions in the table below and restart Logtail. For more information, see Appendix: Restart Logtail.

        For details on the project region, see Service regions.

        Scenario	Network type	<config_region>	<endpoint>
        The server is an Elastic Compute Service (ECS) instance that resides in the same region as the project	Alibaba Cloud internal network	<project region>-intranet	<project region>-intranet.log.aliyuncs.com
        Other scenarios	Public network	<project region>	<project region>.log.aliyuncs.com
        	Transfer acceleration		log-global.aliyuncs.com

Logtail can only upload data to Simple Log Service if the server can connect to the following addresses.

1. The config_server_address field in the ilogtail_config.json file specifies the address, which also has an HTTPS version.

2. http://<project name>.<endpoint>.

   • The project name and region can be viewed as follows.

     

   • <endpoint> refers to the address defined in the data_server_list.endpoint field within the ilogtail_config.json file.

3. Access the service at http://ali-<project region>-sls-admin.<endpoint>. The <endpoint> corresponds to the address defined by the data_server_list.endpoint parameter in the ilogtail_config.json file.

To verify network connectivity, follow these steps:

Linux

1. Log on to the Linux server where Logtail is installed.

2. Run the date command to check the system time.

   Wed Dec 28 06:59:26 UTC 2022

   If the system time is incorrect, take the following actions:

   • Adjust the system time to match the local time.

   • If you cannot adjust the system time, add the following parameter to the ilogtail_config.json file: "enable_log_time_auto_adjust": true. After modifying the file, restart Logtail. For more information, see Appendix: Restart Logtail. For the ilogtail_config.json file path, refer to Step 3: Validate the Logtail startup parameters.

Windows

1. Log on to the Windows server where Logtail is installed.

2. Check the time displayed on the taskbar in the desktop's lower-right corner.

   • Adjust the system time to match the local time.

   • If you cannot adjust the system time, add "enable_log_time_auto_adjust": true to the ilogtail_config.json file. After modifying the file, restart Logtail. For more information, see Appendix: Restart Logtail. For the ilogtail_config.json file path, refer to Step 3: Validate the Logtail startup parameters.

Step 6: Check whether a user identifier exists

Check for a user identifier based on the user identifier file in the specified directory. If no output is returned, ensure a user identifier file exists at the specified path.

• If the user identifier file is missing or invalid in the specified directory, follow these steps to resolve the issue:

  • Linux: Execute cd /etc/ilogtail/users/ && touch <uid> to create a user identifier file, where <uid> is the Alibaba Cloud account ID associated with the project.

  • Windows: Navigate to C:\LogtailData\users\ and create an empty file named <uid>, where <uid> is the Alibaba Cloud account ID associated with the project.

• If a file named after the Alibaba Cloud account ID associated with the project exists in the specified directory, the user identifier is configured correctly.

Step 7: Check whether a custom identifier is configured if you use a custom identifier-based machine group

Verify the presence of a custom identifier on the server by checking the user_defined_id file in the specified directory.

• If no output is returned, confirm the existence of the user_defined_id file or that a custom identifier is configured within it.

  Note

  The path of the user_defined_id file varies based on the operating system:

  • Linux: /etc/ilogtail/user_defined_id

  • Windows: C:\LogtailData\user_defined_id

  • If the user_defined_id file is missing, create one and enter the custom identifier for the machine group. For guidance, see Configuring a custom identifier.

  • If the user_defined_id file is present but lacks a custom identifier or contains an invalid one, add the correct custom identifier to the file. For instructions, see Configuring a custom identifier.

  • If the user_defined_id file includes the custom identifier you specified for the machine group, the custom identifier is correctly configured.

Windows

1. Log on to the Windows server where Logtail is installed.

2. Open the Run dialog and type services.msc.

3. Restart the LogtailDaemon service (Logtail V1.0.0.0 or later) or the LogtailWorker service (Logtail V0.x.x.x).