Aggregate the logs of multiple source Logstores to one destination Logstore - Simple Log Service

Simple Log Service allows you to configure a data transformation job for each Logstore. This way, you can aggregate the logs of multiple source Logstores to one destination Logstore. This topic describes how to aggregate the logs of multiple source Logstores to one destination Logstore and provides common scenarios.

Background information

The services of an information website are distributed around the world. The access logs of users from different information channels are collected and stored in Logstores of different Alibaba Cloud accounts. To aggregate the logs of multiple Logstores to one Logstore for subsequent data query and analysis, you can use the e_output function to transform the logs. For more information, see e_output.

In this topic, the Logstores in the UK (London) region are used to describe how to aggregate the logs of multiple source Logstores to one destination Logstore.

The raw logs in Account 1 are stored in a Logstore named Logstore_1 of a project named Project_1. The project resides in the UK (London) region.

Log 1
request_id: 1
http_host:  example.com
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg

Log 2
request_id: 2
http_host:  aliyundoc.com
http_status:  301
request_method:  POST
request_uri:  /data/data.php

The raw logs in Account 2 are stored in a Logstore named Logstore_2 of a project named Project_2. The project resides in the UK (London) region.

Log 1
request_id: 3
host:  example.edu
status:  404
request_method:  GET
request_uri:  /category/abc/product_id

Log 2
request_id: 4
host:  example.net
status:  200
request_method:  GET
request_uri:  /data/index.html

Transformation requirements
- Export logs whose http_status is 200 in Logstore_1 of Account 1 to Logstore_3 of Account 3.
- Export logs whose http_status is 200 in Logstore_2 of Account 2 to Logstore_3 of Account 3. Before you export the logs, rename host to http_host and status to http_status to keep consistency with the field names in Logstore_1.

Step 1: Configure a data transformation rule for Logstore_1

Go to the data transformation page of Logstore_1 in Account 1. For more information about how to go to the page in the Simple Log Service console, see Create a data transformation job.
On the data transformation page, configure the following data transformation rule. Export logs whose http_status is 200 in Logstore_1 of Account 1 to Logstore_3 of Account 3.
```
e_if(e_match("http_status", "200"), e_output("target_logstore"))
```
Create a data transformation job. In the Storage Destination section, configure the Destination Name, Destination Region, Destination Project, and Target Store parameters based on the following figure. For more information about the Authorization Method parameter, see Create a data transformation job.

Step 2: Configure a data transformation rule for Logstore_2

Go to the data transformation page of Logstore_2 in Account 2. For more information about how to go to the page in the Simple Log Service console, see Create a data transformation job.
On the data transformation page, configure the following data transformation rule. Export logs whose http_status is 200 in Logstore_2 of Account 2 to Logstore_3 of Account 3. Before you export the logs, rename host to http_host and status to http_status to keep consistency with the field names in Logstore_1.
```
e_if(e_match("status", "200"), e_compose(e_rename("status", "http_status", "host", "http_host"), e_output("target_logstore")))
```
Preview results
Create a data transformation job. In the Storage Destination section, configure the Destination Name, Destination Region, Destination Project, and Target Store parameters based on the following figure. For more information about the Authorization Method parameter, see Create a data transformation job.

View aggregation results

Query and analyze logs in Logstore_3 that resides in the UK (London) region. For more information, see Query and analyze logs. Sample logs:

Log 1
request_id: 1
http_host:  example.com
http_status:  200
request_method:  GET
request_uri:  /pic/icon.jpg

Log 2
request_id: 4
http_host:  example.net
http_status:  200
request_method:  GET
request_uri:  /data/index.html