High availability solutions for client data collection - Simple Log Service

This topic describes two high-availability solutions for Simple Log Service to collect data from clients: dual-write and data transformation-based replication plus Logstore switchover.

Background information

In a single-cluster environment, the storage system of Simple Log Service uses a three-replica mechanism to ensure data reliability. Three replicas are provided for all data records. The replicas are stored in different servers of the cluster by using a distributed storage algorithm. The storage system ensures that three replicas are distributed to different physical disks on different servers. If a single hardware device fails, data loss does not occur. The storage system also ensures strong data consistency among the three replicas. Simple Log Service provides high availability even if a single hardware device fails.

However, a cluster may fail due to unexpected reasons in some extreme scenarios. Simple Log Service provides two active geo-redundancy solutions that allow you to collect data from clients when a cluster fails.

Comparison of solutions

Item	Solution 1: dual-write	Solution 2: data transformation-based replication plus Logstore switchover
Deployment	Easy	You must configure data transformation rules and prepare for Logstore switchover.
Client resource usage	The resource usage of Solution 1 is twice the resource usage of Solution 2.	Low
Data consistency	Relatively strong	A specific percentage of data may be duplicated or lost during a switchover.

Network recommendations

Logtail has high requirements for network quality when Logtail collects data from clients. Therefore, we recommend that you use a leased line in cross-region data transfer scenarios.

Solution 1: Dual-write

This solution is used to write client data to two Logstores, such as Logstore A and Logstore B, that reside in different regions in real time.

In normal cases, data in Logstore A and Logstore B is available in real time because the data is written to the Logstores in real time. If an error occurs in Logstore A, the data in Logstore B is still available in real time. If an error occurs in Logstore B, the data in Logstore A is still available in real time.

Procedure:

Create projects and Logstores in the Simple Log Service console.
1. Create a project named Project A and a Logstore named Logstore A in Region A.
2. Create a project named Project B and a Logstore named Logstore B in Region B.

Configure the logtail_config.json file on the server to which your client is connected to allow Logtail to write data in two regions.

For example, if Region A is China (Shanghai) and Region B is China (Beijing), add the config_server_address_list and data_server_list configuration item, use the Simple Log Service endpoint for the China (Beijing) region as the second address, and retain other settings.

{
    ...
    "config_server_address":"logtail.cn-shanghai-intranet.log.aliyuncs.com",
    "config_server_address_list": [
        "cn-shanghai-intranet.log.aliyuncs.com",
        "cn-beijing-intranet.log.aliyuncs.com"
    ],
    "data_server_list": [
        {
            "cluster": "cn-shanghai",
            "endpoint":"cn-shanghai-intranet.log.aliyuncs.com"
        },
        {
            "cluster": "cn-beijing",
            "endpoint":"cn-beijing-intranet.log.aliyuncs.com"
        }
    ],
    "cpu_usage_limit": 2.0,
    ...
}

Create Logtail configurations in the Simple Log Service console.
Important
- If you want to use multiple Logtail configurations to collect logs from a file, you need to specify "accept_multi_config": true in the startup parameters of Logtail or specify {"force_multiconfig": true} in the More Configurations section when you create the Logtail configurations. For more information, see How do I collect multiple copies of logs in a file?
- If you want to change a Logtail configuration in one project, you must also change the Logtail configuration in the other project. Make sure that the configurations are consistent.
1. Create a Logtail configuration in Logstore A of Project A.
2. Create a Logtail configuration in Logstore B of Project B.

Solution 2: Data transformation-based replication plus Logstore switchover

This solution is used to collect data from clients to Logstore A in a region and copy the data to Logstore B in another region in real time based on the data transformation feature. If an error occurs in Logstore A, you must switch to Logstore B and collect the data to Logstore B in real time.

In normal cases, data in Logstore A and Logstore B is available in real time because the data that is written to Logstore A is transformed and then copied to Logstore B in real time. If an error occurs in Logstore A, you must switch to Logstore B. This ensures that the data in Logstore B is available in real time. If an error occurs in Logstore B, the data in Logstore A is still available in real time.

Procedure:

Create projects and Logstores in the Simple Log Service console.
1. Create a project named Project A and a Logstore named Logstore A in Region A.
2. Create a project named Project B and a Logstore named Logstore B in Region B.

Configure the logtail_config.json file on the server to which your client is connected to allow Logtail to write data in two regions.

{
    ...
    "config_server_address":"logtail.cn-shanghai-intranet.log.aliyuncs.com",
    "config_server_address_list": [
        "cn-shanghai-intranet.log.aliyuncs.com",
        "cn-beijing-intranet.log.aliyuncs.com"
    ],
    "data_server_list": [
        {
            "cluster": "cn-shanghai",
            "endpoint":"cn-shanghai-intranet.log.aliyuncs.com"
        },
        {
            "cluster": "cn-beijing",
            "endpoint":"cn-beijing-intranet.log.aliyuncs.com"
        }
    ],
    "cpu_usage_limit": 2.0,
    ...
}

Create a Logtail configuration in Logstore A of Project A in the Simple Log Service console.
Important
If you want to use multiple Logtail configurations to collect logs from a file, you need to configure "accept_multi_config": true in the startup parameters of Logtail or specify {"force_multiconfig": true} in the More Configurations section when you create the Logtail configuration. For more information, see What do I do if I want to use multiple Logtail configurations to collect logs from a log file?
Use Terraform or API to back up the Logtail configuration in Project A and reconfigure the Logtail configuration. Make sure that you can use the reconfigured Logtail configuration to collect data to Logstore B of Project B.
Create a data transformation task to synchronize data between the two Logstores in real time. For more information, see Replicate data from a Logstore and Transmit data across regions.
If an error occurs in Logstore A, import the Logtail configuration that is backed up in Step 4 to Logstore B.