Pull data from one Logstore to enrich log data in another Logstore - Simple Log Service

This topic describes how to use resource functions to pull data from another Logstore to enrich log data in a Logstore.

Background information

A hotel stores the personal information of guests in a Logstore named user_logstore and stores the check-in information of guests in a Logstore named check-in_logstore. The hotel wants to obtain some fields from the check-in_logstore Logstore and concatenate the fields with the fields in the user_logstore Logstore. The hotel can use the res_log_logstore_pull function to pull data from the check-in_logstore Logstore and use the e_table_map or e_search_dict_map function to map data. For more information about the res_log_logstore_pull function, see res_log_logstore_pull. For more information about the e_table_map function, see e_table_map. For more information about the e_search_dict_map function, see e_search_table_map.

Data transformation

Raw data

The Logstore named user_logstore that is used to store personal information

topic:xxx
city:xxx
cid:12345
name:maki


topic:xxx
city:xxx
cid:12346
name:vicky

topic:xxx
city:xxx
cid:12347
name:mary

The Logstore named check-in_logstore that is used to store check-in information

time:1567038284
status:check in
cid:12345
name:maki
room_number:1111

time:1567038284
status:check in
cid:12346
name:vicky
room_number:2222

time:1567038500
status:check in
cid:12347
name:mary
room_number:3333

time:1567038500
status:leave
cid:12345
name:maki
room_number:1111

Transformation rule
Note
The res_log_logstore_pull function allows you to set a time range or a start time for data enrichment.
- If you set a time range in the transformation rule, for example, from_time=1567038284 and to_time=1567038500, data that is received in the specified time range by Simple Log Service is pulled for data enrichment.
- If you set a start time in the transformation rule, for example, from_time="begin", data that is received from the specified time by Simple Log Service is pulled for data enrichment.
For more information about the fields of the res_log_logstore_pull function, see res_log_logstore_pull.
- e_table_map function
  This function maps two log entries by using the cid field. If the value of the cid field in the two log entries equals each other, data mapping succeeds. The room_number field and field value are returned and concatenated with the log entry in the check-in_logstore Logstore.
```
e_table_map(res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, 
        fields=["cid","room_number"],
        from_time="begin",
        ), "cid","room_number")
```
- e_search_table_map function
  This function searches for the cid field whose value is 12346 in the check-in_logstore Logstore, returns the room_number field and its value, and concatenates the field with these fields of the log entries in the user_logstore Logstore.
```
e_search_table_map(res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, 
        fields=["cid","room_number"],
        from_time="begin",
        ), "cid=12346","room_number")
```

Result

e_table_map function

topic:xxx
city:xxx
cid:12345
name:maki
room_nuber:1111

topic:xxx
city:xxx
cid:12346
name:vicky
room_number:2222

topic:xxx
city:xxx
cid:12347
name:mary
room_number:3333

e_search_table_map function

topic:xxx
city:xxx
cid:12346
name:vicky
room_number:2222

Configure a whitelist rule and a blacklist rule to filter data

Configure a whitelist rule
- Transformation rule
  Use the fetch_include_data field to configure a whitelist rule. In this example, the fetch_include_data="room_number:1111" expression is included in the res_log_logstore_pull function. This expression indicates that only the log entries whose room_number field value is 1111 are pulled.
```
res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status"],from_time=1567038284,to_time=1567038500,fetch_include_data="room_number:1111")
```
- Retrieved data
```
status: check in
cid:12345
name:maki
room_number:1111

status:leave
cid:12345
name:maki
room_number:1111
```
Configure a blacklist rule
- Transformation rule
  Use the fetch_exclude_data field to configure a blacklist rule. In this example, the fetch_exclude_data="room_number:1111" expression is included in the res_log_logstore_pull function. This expression indicates that only the log entries whose room_number field value is 1111 are dropped.
```
res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status"],from_time=1567038284,to_time=1567038500,fetch_exclude_data="room_number:1111")
```
- Retrieved data
```
status:check in
cid:12346
name:vicky
room_number:2222


status:check in
cid:12347
name:mary
room_number:3333
```
Configure a blacklist rule and a whitelist rule
- Transformation rule
  If you configure a blacklist rule and a whitelist rule, the blacklist rule is applied first and then the whitelist rule. In this example, the fetch_exclude_data="time:1567038285",fetch_include_data="status:check in" expression is included in the res_log_logstore_pull function. This expression indicates that the log entries whose time field value is 1567038285 dropped first and then the log entries whose status field value is check in are pulled.
```
res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status"],from_time=1567038284,to_time=1567038500,fetch_exclude_data="time:1567038285",fetch_include_data="status:check in")
```
- Retrieved data
```
status:check in
cid:12345
name:maki
room_number:1111


status:check in
cid:12346
name:vicky
room_number:2222


status:check in
cid:12347
name:mary
room_number:3333
```

Enable primary key maintenance to pull data from destination Logstores

To discard data before transforming it, you can enable the primary key maintenance feature. For example, you want to pull the check-in data of customers who have checked in but not checked out from the Logstore named check-in_logstore. If a pulled log entry of a customer includes the status:leave field, the customer has checked out. You can set the primary_keys parameter of the res_log_logstore_pull function and the log entry is not transformed.

Note

You can set a single field as the value of the primary_keys parameter. The field must exist in the fields field.
Before you enable the primary key maintenance feature, make sure that the Logstore from which you want to pull data has only one shard.
If you enable the primary key maintenance feature, you cannot set the delete_data field to None.

Transformation rule

res_log_logstore_pull(endpoint, ak_id, ak_secret, project, logstore, ["cid","name","room_number","status","time"],from_time=1567038284,to_time=None,primary_keys="cid",delete_data="status:leave")

Retrieve data
The status:leave field in the preceding log entry indicates that the customer whose name is maki has checked out. Therefore, this log entry is not transformed.
```
time:1567038284
status:check in
cid:12346
name:vicky
room_number:2222

time:1567038500
status:check in
cid:12347
name:mary
room_number:3333
```