Build dictionaries and tables for data enrichment - Simple Log Service

Dictionaries and tables are two major types of data structures used for data enrichment. This topic describes common methods for building dictionaries and tables and compares the advantages and disadvantages of different building methods.

Build dictionaries

Build a dictionary directly

e_dict_map({"400": "error", "200": "ok", "*": "other"}, "status", "message")

Build a dictionary from data configuration items
```
e_dict_map(res_local("http_code_map"), "status", "message")
```
where, http_code_map is an advanced task configuration item. Its values are as follows:
```
{"400": "error", "200": "ok", "*": "other"}
```
Build a dictionary from a table
Use the tab_to_dict function to build a dictionary from a table. Details on how to build a table are available in later part of this topic.
```
e_dict_map(tab_to_dict(tab_parse_csv("status_code,status_info\n400,error\n200,ok\n*,other"), "status_code", "status_info"), "status", "message")
```

Build a dictionary by using the dct_make function

e_dict_map(dct_make("400", "error", "200", "ok", "*", "other"), "status", "message")

Build a dictionary by using another expression
```
e_dict_map(json_parse(v("http_code_map")), "status", "message")
```
where, the data mappings are obtained from the http_code_map field in source logs.

The following table compares these dictionary building methods.

Method	Advantage	Disadvantage
Build a dictionary directly	This method is intuitive, simple, and easy to use.	The rules will be lengthy if too much content is involved. In addition, the built dictionary is static and inflexible.
Build a dictionary from data configuration items	We recommend that you use this method if the dictionary contains a large amount of data and is frequently modified. This method features easy maintenance.	A dictionary built in this method is not scalable or reusable across tasks. In addition, it does not support automatic update.
Build a dictionary from a table	Featuring flexible maintenance, this method is used in advanced scenarios.	You need to build and maintain a table for building a dictionary, making the configuration process complicated.
Build a dictionary by using the dct_make function	You can build a dictionary based on dynamic logic in specific scenarios.	This method is relatively advanced and difficult to maintain.
Build a dictionary by using another expression	This method dynamically extracts data mappings from fields in log events. It applies in specific scenarios.	This method is relatively advanced and difficult to maintain.

Build tables

Build a table from a text file

e_table_map(tab_parse_csv("city,name,age\nshanghai,aliyun,10\ncity:nanjing,Maki,18"), "name",["city", "age"])

Build a table from data configuration items
```
e_search_table_map(tab_parse_csv(res_local("table_info")), "name",["city", "age"])
```
where, table_info is a task configuration item in data transformation rules. Its values are as follows:
```
content,name,age
shanghai,aliyun,10
nanjing,Maki,18
```

Build a table from an ApsaraDB for Relational Database Service (RDS) database

e_table_map(tab_parse_csv(res_rds_mysql(...database="db", table="city")), "name",["city", "age"])

The city table in the RDS database contains the following information:

content,name,age
shanghai,aliyun,10
nanjing,Maki,18

Build a table from another Logstore

e_table_map(res_log_logstore_pull(..., project="project_name", logstore="logstore_name", fields=["city","name","age"]),, "name",["city", "age"])

The specified Logstore contains the following log events:

"Log 1"
{
  "city": "shanghai",
  "name": "aliyun",
  "age": "10"
}
"Log 2"
{
  "city": "city:nanjing and data > 100",
  "name": "Maki",
  "age": "18"
}

The following table compares these table building methods.

Method	Advantage	Disadvantage
Build a table from a text file	This method is intuitive, simple, and easy to use.	The rules will be lengthy if too much content is involved. A table built in this method is difficult to maintain, scale up, or reuse.
Build a table from data configuration items	We recommend that you use this method if the table contains a large amount of data and is frequently modified. This method features easy maintenance.	A table built in this method is not scalable or reusable across tasks. In addition, it does not support automatic update.
Build a table from an ApsaraDB for RDS database	We recommend that you use this method if the table contains a large amount of data and is frequently modified. This method features easy maintenance. This method supports automatic update. A table built in this method can be reused across tasks.	You need to connect to an external ApsaraDB for RDS database for building a table, making the configuration process complicated.
Build a table from another Logstore	This method reads data in real time. Featuring flexible maintenance, this method is used in advanced scenarios.	You need to connect to another Logstore for building a table, making the configuration process complicated.