Use Logtail to collect logs across regions - Simple Log Service

This topic describes how to use Logtail to collect logs from Linux servers within the same Alibaba Cloud account across regions.

Solution overview

A company deploys its website application on an Elastic Compute Service (ECS) instance that resides in Region A and deploys its Simple Log Service project in Region B. The company wants to collect logs from the ECS instance that resides in Region A to its Simple Log Service project in Region B by using a Logtail configuration. Procedure:

Step 1: Install Logtail on the ECS instance in Region A
Step 2: Configure a custom identifier for the ECS instance in Region A
Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B
Step 4: Create a Logtail configuration in the Simple Log Service project in Region B

The following figure shows the configuration procedure.

Prerequisites

A project and a Logstore are created. For more information, see Create a project and Create a Logstore.

Step 1: Install Logtail on the ECS instance in Region A

Network transmission

Transmission mode	Scenario
Internet	The Alibaba Cloud ECS instance and the Simple Log Service project reside in different regions. The server that is used is from a third-party cloud service provider or in a data center.
Transfer acceleration	The server that is used is from a third-party cloud service provider outside China or in a data center outside China. Data transfer over the Internet may cause high network latency and unstable connections. In this case, we recommend that you use the transfer acceleration feature to transfer logs. For more information, see Use the transfer acceleration feature.

Log on to the ECS instance that resides in Region A and select a Logtail installation script based on your network conditions. For more information, see Network transmission. For more information about the Linux operating systems supported by Logtail, see Limits.

Internet

Obtain the value of the Simple Log Service parameter for the region in which the Simple Log Service project resides. Replace the ${region_id} parameter with the obtained value and run the installation command.

Important

For more information about the value of the ${region_id} parameter for each region, see Supported regions. For example, the value of the ${region_id} parameter for the China (Hangzhou) region is cn-hangzhou.

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-internet

Transfer acceleration

Important

wget http://logtail-release-${region_id}.oss-${region_id}.aliyuncs.com/linux64/logtail.sh -O logtail.sh; chmod +x logtail.sh; ./logtail.sh install ${region_id}-acceleration

Step 2: Configure a custom identifier for the ECS instance in Region A

Log on to the ECS instance on which Logtail is installed. Create a custom identifier file named user_defined_id in a specified directory and configure a custom identifier.

If you want to configure user-defined-linux-1 as a custom identifier for a Linux server, run the following command:

echo "user-defined-linux-1" > /etc/ilogtail/user_defined_id

Step 3: Create a custom identifier-based machine group in the Simple Log Service project in Region B

Log on to the Simple Log Service console. In the Projects section, click the project that you want to manage.
In the left-side navigation pane, choose Resources > Machine Groups. In the Machine Groups list, choose > Create Machine Group.

In the Create Machine Group panel, configure parameters and click OK. The following table describes the parameters.

Parameter	Description
Name	The name of the machine group. The name must meet the following requirements: The name can contain only lowercase letters, digits, hyphens (-), and underscores (_). The name must start and end with a lowercase letter or a digit. The name must be 2 to 128 characters in length. Important After you create a machine group, you cannot change the name of the machine group. Proceed with caution.
Machine Group Identifier	The identifier type of the machine group. In this example, select Custom Identifier.
Machine Group Topic	Optional. The topic of the machine group. The topic is used to identify the logs that are generated by different servers. For more information, see Log topics.
Custom Identifier	The custom identifier. In this example, enter `user-defined-linux-1`.

In the Machine Groups list, click the created machine group. On the Machine Group Configurations page, you can check the configuration of the machine group and the server status.
- If the value in the Heartbeat column for a server is OK, Logtail on the server is connected to Simple Log Service.
- If the value in the Heartbeat column for a server is FAIL, an error occurs. For more information about how to troubleshoot the error, see How do I troubleshoot an error related to a Logtail machine group in a host environment?
  Important
  Approximately 2 minutes are required to create a machine group. If the machine group is not created, the heartbeat status of the machine group is FAIL. In this case, click Refresh 2 minutes later.

Step 4: Create a Logtail configuration in the Simple Log Service project in Region B

Important

Ports 80 and 443 are enabled for the server on which Logtail is installed to allow outbound traffic. If the server is an ECS instance, you can reconfigure the related security group rules to enable the ports. For more information about how to configure a security group rule, see Add a security group rule.
The server from which you want to collect logs continuously generates logs. Logtail collects only incremental logs. If a log file on your server is not updated after a Logtail configuration is delivered and applied to the server, Logtail does not collect logs from the file. For more information, see Read log files.
If you want to collect logs from historical log files, see Import historical logs from log files.

For more information about the common methods of log collection, see the following best practices:

References

If the preview page is blank or the query page displays no data when you use Logtail to collect logs, you can troubleshoot the issue by following the instructions provided in What do I do if errors occur when I use Logtail to collect logs? If you use Logtail to collect logs, errors may occur. For example, regular expressions may fail to be parsed, invalid file paths may exist, and traffic may exceed the processing capabilities of shards. You can view Logtail collection errors by following the instructions provided in How do I view Logtail collection errors? For more information about the common errors in data collection, see How do I troubleshoot the common errors that may occur when Simple Log Service collects logs?.
By default, you can use only one Logtail configuration to collect logs from a log file. If you want to collect multiple copies of logs in a file, you can follow the instructions provided in How do I collect multiple copies of logs in a file?
For more information about how to collect logs from servers in a corporate intranet to Simple Log Service, see Collect logs from servers in a corporate intranet.