Use Logtail to collect logs across regions - Simple Log Service

This topic describes how to use Logtail to collect logs from Linux servers within the same Alibaba Cloud account across regions.

Solution overview

A company deploys its website application on an Elastic Compute Service (ECS) instance that resides in Region A and deploys its Simple Log Service project in Region B. The company wants to collect logs from the ECS instance that resides in Region A to its Simple Log Service project in Region B by using a Logtail configuration. Procedure:

Step 1: Install Logtail on the ECS instance in Region A
Step 2: Configure a custom identifier for the ECS instance in Region A
Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B
Step 4: Create a Logtail configuration in the Simple Log Service project in Region B

The following figure shows the configuration procedure.

Prerequisites

A project and a Logstore are created. For more information, see Create a project and Create a Logstore.

Step 1: Install Logtail on the ECS instance in Region A

Network transmission

Transmission mode	Scenario
Internet	The Alibaba Cloud ECS instance and the Simple Log Service project reside in different regions. The server that is used is from a third-party cloud service provider or in a data center.
Transfer acceleration	The server that is used is from a third-party cloud service provider outside China or in a data center outside China. Data transfer over the Internet may cause high network latency and unstable connections. In this case, we recommend that you use the transfer acceleration feature to transfer logs. For more information, see Use the transfer acceleration feature.

Log on to the ECS instance that resides in Region A and select a Logtail installation script based on your network conditions. For more information, see Network transmission. For more information about the Linux operating systems supported by Logtail, see Limits.

Internet

Obtain the value of the Simple Log Service parameter for the region in which the Simple Log Service project resides. Replace the ${region_id} parameter with the obtained value and run the installation command.

Important

For more information about the value of the ${region_id} parameter for each region, see Supported regions. For example, the value of the ${region_id} parameter for the China (Hangzhou) region is cn-hangzhou.

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-internet

Transfer acceleration

Important

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-acceleration

Step 2: Configure a custom identifier for the ECS instance in Region A

Create a custom identifier file named user_defined_id in a specified directory and configure a custom identifier.

Important

A machine group cannot include both Linux and Windows servers. Do not configure the same custom identifier for Linux and Windows servers.
You can configure one or more custom identifiers for a single server and separate the identifiers with line feeds.
The following list describes the paths to the custom identifier file:
- Linux: /etc/ilogtail/user_defined_id
- Windows: C:\LogtailData\user_defined_id
On a Linux server, if the /etc/ilogtail/ directory or the /etc/ilogtail/user_defined_id file does not exist, you must create the directory or file. On a Windows server, if the C:\LogtailData directory or the C:\LogtailData\user_defined_id file does not exist, you must also create the directory or file.

Linux

To configure a custom identifier for a Linux server, run the following command:
```
echo "user-defined-linux-1" > /etc/ilogtail/user_defined_id
```
To check whether the custom identifier is configured, run the following command:
```
[root@test-host ~]# cat /etc/ilogtail/user_defined_id
user-defined-linux-1
```
By default, after you create, delete, or modify the user_defined_id file, the new settings take effect within 1 minute. If you want the new settings to immediately take effect, run the following commands to restart Logtail:
```
/etc/init.d/ilogtaild stop
/etc/init.d/ilogtaild start
```

Windows

To configure a custom identifier for a Windows server, enter user-defined-windows-1 in the C:\LogtailData\user_defined_id file and save the file.
To check whether the custom identifier is configured, open and view the C:\LogtailData\user_defined_id file.
By default, after you create, delete, or modify the user_defined_id file, the new settings take effect within 1 minute. If you want the new settings to immediately take effect, perform the following operations to restart Logtail:
1. Choose Start Menu > Control Panel > Administrative Tools > Services.
2. In the Services dialog box, select the service that you want to manage.
3. For Logtail V0.x.x.x, select LogtailWorker.
4. For Logtail V1.0.0.0 or later, select LogtailDaemon.
5. Right-click the service and select Restart for the settings to take effect.

Container

The custom identifier file is stored in a Logtail container. The file path is specified in the ALIYUN_LOGTAIL_USER_DEFINED_ID environment variable of the Logtail container. You can run the docker inspect ${logtail_container_name} | grep ALIYUN_LOGTAIL_USER_DEFINED_ID command to view the file path.

Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B

Log on to the Simple Log Service console. In the Projects section, click the project that you want to manage.
In the left-side navigation pane, choose Resources > Machine Groups. In the Machine Groups list, choose > Create Machine Group.

In the Create Machine Group panel, configure parameters and click OK. The following table describes the parameters.

Parameter	Description
Name	The name of the machine group. The name must meet the following requirements: The name can contain only lowercase letters, digits, hyphens (-), and underscores (_). The name must start and end with a lowercase letter or a digit. The name must be 2 to 128 characters in length. Important After you create a machine group, you cannot change the name of the machine group. Proceed with caution.
Machine Group Identifier	The identifier type of the machine group. Select IP Address.
Machine Group Topic	Optional. The topic of the machine group. The topic is used to identify the logs that are generated by different servers. For more information, see Log topics.
IP Address	The IP address. Enter the IP address that is obtained in Step 1. Important If you want to add multiple servers to a machine group, we recommend that you manually enter the IP addresses of the servers and separate the IP addresses with line feeds. A machine group cannot include both Linux and Windows servers. Do not add the IP addresses of Linux and Windows servers to the same machine group.

In the Machine Groups list, click the created machine group. On the Machine Group Configurations page, you can check the configuration of the machine group and the server status.
- If the value in the Heartbeat column for a server is OK, Logtail on the server is connected to Simple Log Service.
- If the value in the Heartbeat column for a server is FAIL, an error occurs. For more information about how to troubleshoot the error, see How do I troubleshoot an error related to a Logtail machine group in a host environment?
  Important
  Approximately 2 minutes are required to create a machine group. If the machine group is not created, the heartbeat status of the machine group is FAIL. In this case, click Refresh 2 minutes later.

Step 4: Create a Logtail configuration in the Simple Log Service project in Region B

Important

Ports 80 and 443 are enabled for the server on which Logtail is installed to allow outbound traffic. If the server is an ECS instance, you can reconfigure the related security group rules to enable the ports. For more information about how to configure a security group rule, see Add a security group rule.
The server from which you want to collect logs continuously generates logs. Logtail collects only incremental logs. If a log file on your server is not updated after a Logtail configuration is delivered and applied to the server, Logtail does not collect logs from the file. For more information, see Read log files.
If you want to collect logs from historical log files, see Import historical logs from log files.

For more information about the common methods of log collection, see the following best practices:

References

If the preview page is blank or the query page displays no data when you use Logtail to collect logs, you can troubleshoot the issue by following the instructions provided in What do I do if errors occur when I use Logtail to collect logs? If you use Logtail to collect logs, errors may occur. For example, regular expressions may fail to be parsed, invalid file paths may exist, and traffic may exceed the processing capabilities of shards. You can view Logtail collection errors by following the instructions provided in How do I view Logtail collection errors? For more information about the common errors in data collection, see How do I troubleshoot the common errors that may occur when Simple Log Service collects logs?.
By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs in a file, you can follow the instructions provided in How do I collect multiple copies of logs in a file?
For more information about how to collect logs from servers in a corporate intranet to Simple Log Service, see Collect logs from servers in a corporate intranet.