Simple Log Service:Language introduction

DSL for Simple Log Service provides approximately 30 global processing functions. You can configure the parameters of global processing functions to control processing operations. Global processing functions accept the results of expression functions as parameters. Flow control functions are a type of global processing function and can be used together with expression functions and the following types of global processing functions:

• Flow control functions

  • You can control processes based on conditions by using functions such as e_if_else, e_if, e_switch, and e_compose.

  • You can use simple search functions, such as e_search, to process different types of logs in a flexible manner.

• Event processing functions

  You can discard, retain, split, write, and replicate events.

• Field processing functions

  You can retain, delete, and rename fields.

• Value extraction functions

  • You can extract values or key-value pairs from fields based on regular expressions, Grok patterns, syslog protocols, quotes, key-value pair delimiters, and delimiters such as commas (,), vertical bars (|), and tabs (\t).

  • You can extract and enrich JSON data.

• Mapping and enrichment functions

  • You can map or search for data based on a dictionary or a table.

  • You can obtain information about a dimension table that is used to enrich data from resources such as rule configurations, external OSS buckets, and ApsaraDB RDS for MySQL instances.

  • You can use a function to automatically refresh external resources based on full or incremental change logs.

• Value-added content functions

  You can enrich the information about some log fields. For example, you can obtain threat intelligence for an IP address and store the threat intelligence to log fields for log analysis.

DSL for Simple Log Service provides more than 200 built-in expression functions to convert events or affect the results of the global processing functions. The expression functions are suitable for most data transformation scenarios. DSL for Simple Log Service provides the following expression functions:

• Event check functions

  DSL for Simple Log Service provides a condition-based filtering mechanism that uses Lucene-like syntax, complete regular expressions, strings, generic characters, numeric value comparison, and logical operators such as AND, OR, and NOT.

• Operator functions

  You can extract, control, and compare field values. You can also perform container evaluation and operations on multiple fields.

• Conversion functions

  You can convert the values of basic data types. You can also convert numbers, dictionaries, and lists.

• Arithmetic functions

  You can perform basic, multi-value, and mathematical calculations. You can also perform operations based on mathematical parameters.

• String functions

  You can encode, decode, sort, reverse, replace, normalize, search, evaluate, truncate, and format multiple fields. You can also perform evaluation based on character sets.

• Date and time functions

  You can convert date and time values. You can obtain date and time attributes, date and time values, UNIX timestamps, and date and time strings. You can also modify and compare date and time values.

• Regular expression functions

  You can extract, match, evaluate, replace, and truncate fields.

• Grok function

  DSL for Simple Log Service provides more than 400 built-in Grok patterns. Grok patterns can be replaced.

• Structured data functions

  You can extract and filter JSON, Protobuf, and XML data.

• IP address parsing functions

  You can parse IP addresses and convert data.

• Encoding and decoding functions

  You can encode and decode text in the SHA1, SHA256, SHA512, MD5, HTML, URL, or Base64 format.