Simple Log Service:Data transformation syntax

The DSL for Simple Log Service is compatible with Python. In standard mode, the DSL can be regarded as a subset of Python. Except for basic data structures and expressions, other syntax rules are orchestrated by using functions. If you want to use user-defined functions (UDFs), submit a ticket.

e_if(op_ge(op_len(json_parse(v("x"))), 1), e_set("x0", lst_get(v("x"), 0)))
e_if(op_ge(op_len(json_parse(v("x"))), 2), e_set("x1", lst_get(v("x"), 1)))

In standard mode of the DSL for Simple Log Service, all operations are performed by calling functions. The DSL provides more than 200 built-in functions, which are categorized into global processing functions and expression functions.

• Global processing functions

  Global processing functions are used to receive, process, and return logs. Only global processing functions can be used to construct each step in a transformation rule.

• Expression functions

  Expression functions are commonly used to receive specific parameters and return specific values. Expression functions can be combined and passed to global processing functions as parameters to define more flexible logic.

The following table describes global processing functions and expression functions.

Global processing functions are used to receive, process, and return logs.

The following syntax is used:

Global Processing Function 1(..Parameters....)
Global Processing Function 2(..Parameters....)
Global Processing Function 3(..Parameters....)
Global Processing Function 4(..Parameters....)

Global processing functions can be further categorized into flow control functions and event processing functions. The following table describes the functions.

Transformation logic:

• Basic processing

  The data transformation feature reads streaming data from a source Logstore and sends each log in a dictionary structure to specific functions. Then, the feature runs the functions that are specified in the transformation rule in sequence to process the events and writes the transformation results to specified destination Logstores.

  Note

  All fields and values of a log are sent as strings. For example, the raw log {"__time__": "1234567", "__topic__": "", "k1": "test"} is processed by the e_set("f1", 200) function. The function adds the f1 field whose value is set to 200 to the raw log. Then, the raw log is transformed into {"__time__": "1234567", "__topic__": "", "k1": "test", "f1": "200"}. In this log, the f1 field and the value 200 are strings.

  The event processing functions specified in a transformation rule are called in sequence. Each function receives and processes a log, and then returns a processed log.

  For example, the e_set("type", "test") function adds the type field whose value is set to test to a log. The next function receives and processes the processed log.

• Condition evaluation

  • e_if: You can call the e_if function to add conditional expressions to process logs. If a log does not meet the specified condition, the corresponding operation is skipped. The e_if function implements the if logic.

    For example, the e_if(e_match("status", "200"), e_regex("data", "ret: \d+", "result")) function checks whether the value of the status field is 200. If the value is 200, the function extracts the result field from the data field by using the specified regular expression. Otherwise, no operation is performed.

  • e_if_else: This function works in a manner similar to the if_else function.

• Processing termination

  • A step in a transformation rule may return no log. This indicates that the related log is deleted.

    For example, the e_if(str_islower(v("result")), e_drop()) function is used to check whether the value of the result field in a log is a string that consists of only lowercase characters. If the condition is evaluated to true, the log is discarded and the subsequent steps are not performed on this log. The system automatically processes the next log.

  • If a log is written to a destination Logstore, the processing is terminated. For example, if the e_output function is used to write a log to a destination Logstore and delete the log, the subsequent steps are not performed on this log.

    Note

    The e_coutput function copies the output log and the subsequent steps are performed on this log.

• Log splitting for parallel processing

  A step in a transformation rule may return multiple logs. This indicates that the related log is split.

  For example, the e_split(data) function splits a log into two logs based on the value of the data field. If the value of the data field in the log is "abc, xyz", the log is split into two logs. In one log, the value of thedata field is abc. In the other log, the value of the data field is xyz.

  The logs that are generated after splitting are processed in the subsequent steps.

In addition to global processing functions, the DSL for Simple Log Service provides 200 expression functions that are used to receive specific parameters and return specific values. You can call an expression function or a combination of expression functions in a global processing function. The following syntax is used:

Global Processing Function 1(Expression Function 1(...), ...)
Global Processing Function 2(..., Expression Function 2(...), Expression Function 3(...), ...)

Expression functions can be categorized into event check functions, resource functions, control functions, and other expression functions. The following table describes the functions.