This topic describes the syntax and parameters of event processing functions. This topic also provides examples on how to use the functions.

Functions

Category Function Description
Event processing e_drop Discards a log if a specified condition is met.

This function can be used together with other functions. For more information, see Replicate and distribute data.

e_keep Retains a log if a specified condition is met.
Both the e_keep and e_drop functions can be used to discard logs. The difference is that the e_keep function discards logs if a specified condition is not met, whereas the e_drop function discards logs if a specified condition is met. 
# The following four transformation rules are equivalent:
e_if_else(e_search("f1==v1"), KEEP, DROP)
e_if_else(e_search("not f1==v1"), DROP) 
e_keep(e_search("f1==v1"))
e_drop(e_search("not f1==v1"))

# The following transformation rules are invalid:
e_if(e_search("..."), KEEP)   
e_keep()

This function can be used together with other functions. For more information, see Transform complex JSON data.

Event splitting e_split Splits a log into multiple logs based on the value of a specified field. You can also use the JMESPath expression to extract the value of the field, and then split the log.

This function can be used together with other functions. For more information, see Transform complex JSON data.

Event generation e_output and e_coutput Writes a log to a specified Logstore. You can specify the topic, source, tags and MD5 hash keys of a shard for the log.
  • e_output: writes a log to a specified Logstore. The subsequent transformation rules are not executed for the log.
  • e_coutput: writes a log to a specified Logstore. The subsequent transformation rules are executed for the log.

This function can be used together with other functions. For more information, see Aggregate data from multiple source Logstores.

Conversion of events to metrics e_to_metric Converts logs to metrics that can be stored in a Metricstore.
Note After logs are converted to metrics, you must select a Metricstore to save the metrics.
The following example shows typical metrics:
__labels__:host#$#myhost
__name__:rt
__time_nano__:1614739608000000000
__value__:123.0
For more information, see Metric.

This function can be used together with other functions. For more information, see Convert logs to metrics.

e_drop

The e_drop function discards a log if a specified condition is met.

  • Syntax

    e_drop(condition=True)

    The identifier DROP is supported. The identifier DROP is equivalent to the e_drop() function.

  • Parameters

    Parameter Type Required Description
    condition Bool No Default value: True. In most cases, one condition is passed to a function.

  • Response

    If the specified condition is met, the log is discarded and None is returned. If the specified condition is not met, the log is returned.

  • Examples

    • Example 1: If the value of the __programe__ field in a log is access, discard the log. Otherwise, retain the log.
      • Raw log:
        __programe__: access
age:  18
content:  123
name:  maki

__programe__: error
age:  18
content:  123
name:  maki
      • Transformation rule:
        e_if(e_search("__programe__==access"), DROP)
      • Result:

        The log in which the value of the __programe__ field is access is discarded. The log in which the value of the __programe__ field is error is retained. 

        __programe__: error
age:  18
content:  123
name:  maki
    • Example 2: If the specified condition evaluates to True, discard the log.
      • Raw log:
        k1: v1
k2: v2
k3: k1
      • Transformation rule:
        e_drop(e_search("k1==v1"))
      • Result:

        The log is discarded because the k1==v1 condition evaluates to True.

    • Example 3: If the specified condition evaluates to False, retain the log.
      • Raw log:
        k1: v1
k2: v2
k3: k1
      • Transformation rule:
        e_drop(e_search("not k1==v1"))
      • Result:
        k1: v1
k2: v2
k3: k1
    • Example 4: If no conditions are specified, use True, which indicates that the log is discarded.
      • Raw log:
        k1: v1
k2: v2
k3: k1
      • Transformation rule:
        e_drop()
      • Result:

        The log is discarded.

  • References

    This function can be used together with other functions. For more information, see Replicate and distribute data.

e_keep

The e_keep function retains a log if a specified condition is met.

  • Syntax

    e_keep(condition=True)

    The identifier KEEP is supported. The identifier KEEP is equivalent to the e_keep() function.

  • Parameters

    Parameter Type Required Description
    condition Bool No Default value: True. In most cases, one condition is passed to a function.

  • Response

    If the specified condition is met, the log is returned. If the specified condition is not met, the log is discarded.

  • Examples

    • Example 1: If the value of the __programe__ field in a log is access, retain the log. Otherwise, discard the log.
      • Raw log:
        __programe__: access
age:  18
content:  123
name:  maki
__programe__: error
age:  18
content:  123
name:  maki
      • Transformation rule:
        e_keep(e_search("__programe__==access"))
# Equivalent to:
e_if(e_search("not __programe__==access"), DROP) 
# Equivalent to:
e_if_else(e_search("__programe__==access"), KEEP, DROP)
      • Result:

        The log in which the value of the __programe__ field is access is retained. 

        __programe__: access
age:  18
content:  123
name:  maki
    • Example 2: If the specified condition evaluates to True, retain the log.
      • Raw log:
        k1: v1
k2: v2
k3: k1
      • Transformation rule:
        e_keep(e_search("k1==v1"))
      • Result:
        k1: v1
k2: v2
k3: k1
    • Example 3: If the specified condition evaluates to False, discard the log.
      • Raw log:
        k1: v1
k2: v2
k3: k1
      • Transformation rule:
        e_keep(e_search("not k1==v1"))
      • Result:

        The log is discarded.

    • Example 4: Pass the value False to the e_keep function.
      • Raw log:
        k1: v1
k2: v2
k3: k1
      • Transformation rule:
        e_keep(False)
      • Result:

        The log is discarded.

  • References

    This function can be used together with other functions. For more information, see Transform complex JSON data.

e_split

The e_split splits a log into multiple logs based on the value of a specified field. You can also use the JMESPath expression to extract the value of the field, and then split the log.

  • Syntax

    e_split(Field name, sep=',', quote='"', lstrip=True, jmes=None, output=None)
    Splitting rules:
    1. If you configure the jmes parameter, Log Service converts the values of the log field to a JSON list, uses the JMESPath expression to extract the values from the JSON list, and then uses these values in the next operation. If you do not configure the jmes parameter, Log Service uses the values of the log field in the next operation.
    2. If the values obtained from the previous operation is a list or a string that represents a JSON list, Log Service splits the log based on the list. Otherwise, Log Service parses the values into CSV values based on the sep, quote, or lstrip parameter. Then, Log Service splits the log based on the parsed values.

  • Parameters

    Parameter Type Required Description
    Field name String Yes The name of the field that you use to split a log. For more information about how to specify special field names, see Event structure and fields.
    sep String No The delimiter that you use to separate values.
    quote String No The quote that you use to enclose a value.
    lstrip String No Specifies whether to remove the spaces to the left of a value. Default value: True.
    jmes String No The JMESPath string that you use to convert the values of the field to a JSON object and extract the values from the JSON object. Log Service splits the log based on the JSON object.
    output String No The new name of the field, which overwrites the existing name by default.

  • Response

    A list of logs is returned. The values of fields in the returned logs are the same as the values of the fields in the raw log.

  • Examples

    • Raw log:
      __topic__:
age:  18
content:  123
name:  maki

__topic__:
age:  18
content:  123
name:  maki
    • Transformation rule:
      e_set("__topic__", "V_SENT,V_RECV,A_SENT,A_RECV")
e_split("__topic__")
    • Result:
      __topic__:  A_SENT
age:  18
content:  123
name:  maki

__topic__:  V_RECV
age:  18
content:  123
name:  maki

...

  • References

    This function can be used together with other functions. For more information, see Transform complex JSON data.

e_output and e_coutput

The e_output and e_coutput functions write a log to a specified Logstore. You can specify the topic, source, and tags for the log.

  • Syntax

    e_output(name=None, project=None, logstore=None, topic=None, source=None, tags=None, hash_key_field=None, hash_key=None)
e_coutput(name=None, project=None, logstore=None, topic=None, source=None, tags=None, hash_key_field=None, hash_key=None)

    During preview, the log is written to a Logstore named internal-etl-log instead of the specified Logstore. The first time that you preview data transformation results, Log Service automatically creates a dedicated Logstore named internal-etl-log in the current project. You cannot modify the configurations of this Logstore or write other data to the Logstore. You are not charged for this Logstore.

  • Parameters

    Note If you configure the name, project, and logstore parameters in the e_output or e_coutput function and specify the project and Logstore in the Create Data Transformation Rule panel, the configurations in the e_output or e_coutput function take precedence. The following list describes the configurations:
    • If you configure only the name parameter in the e_output or e_coutput function, the transformation result is sent and stored in the Logstore that corresponds to the name parameter.
    • If you configure only the project and logstore parameters in the e_output function, the transformation result is sent and stored in the Logstore specified in the e_output function.

      If you use an AccessKey pair to authorize data transformation, the AccessKey pair of the current logon account is used to transform data.

    • If you configure the name, project, and logstore parameters in the e_output function, the transformation result is sent and stored in the Logstore specified in the e_output function.

      If you use an AccessKey pair to authorize data transformation, the AccessKey pair specified in the storage destination is used to transform data.

    Parameter Type Required Description
    name String No The name of the storage destination. Default value: None.
    project String No The existing project to which the log is written.
    logstore String No The existing Logstore to which the log is written.
    topic String No The new topic of the log.
    source String No The new source of the log.
    tags Dict No The new tags of the log. The tags are in the dictionary format.
    Note You do not need to prefix keywords with __tag__:.
    hash_key_field String No The name of the field that is used for hashing. The log is written to a shard of the storage destination that you specify based on the hash value of the field.
    Note If the log does not contain the field that you specify, the log is randomly written to a shard of the storage destination that you specify in load balancing mode.
    hash_key String No The hash value. The log is written to a shard of the storage destination that you specify based on the hash value.
    Note The hash_key_field parameter has a higher priority than the hash_key parameter. If the hash_key_field parameter is configured in a transformation rule, the hash_key parameter does not take effect.
    • Default storage destination
      To use the e_output or e_coutput function, you must configure a default storage destination in the Create Data Transformation Rule panel. By default, Log Service uses the storage destination labelled 1 as the default storage destination. In the following figure, the transformation result is shipped to the Logstores that correspond to target_01, target_02, and target_03. Data that is not discarded during transformation is stored in the Logstore that corresponds to the default storage destination named target_00. Default storage destination
    • Advanced Parameter Settings
      If the project or Logstore that you specify in the e_output or e_coutput function does not exist, you can specify key-value pairs in the Advanced Parameter Settings section of the Create Data Transformation Rule panel. You can set a key to config.sls_output.failure_strategy and the value of the key to {"drop_when_not_exists":"true"} to skip logs. The skipped logs are discarded and reported as warning logs. If you do not specify key-value pairs in the Advanced Parameter Settings section, your data transformation job is suspended until the project or Logstore that you specify is created.
      Warning If the specified project or Logstore does not exist and you specify key-value pairs in the Advanced Parameter Settings section to skip logs, the skipped logs are discarded. Proceed with caution.
      Advanced Parameter Settings
    • Result:
      • e_output: writes a log to a specified Logstore. The subsequent transformation rules are not executed for the log.
      • e_coutput: writes a log to a specified Logstore. The subsequent transformation rules are executed for the log.

  • Examples

    • Example 1: Evaluate the value of the k2 field in a log against the regular expression. If the value meets the regular expression, write the log to the Logstore specified in target2 and set topic to topic1.
      • Raw log:
        __topic__:
k1: v1
k2: v2
x1: v3
x5: v4
      • Transformation rule:

        The e_drop() function deletes the data that does not meet the condition of the e_if() function. If you do not add the e_drop() function to the transformation rule, the data that does not meet the condition of the e_if() function is shipped to the default storage destination. 

        e_if(e_match("k2", r"\w+"), e_output(name="target2", source="source1", topic="topic1"))
e_drop()
      • Result:
        __topic__:  topic1
k1: v1
k2: v2
x1: v3
x5: v4
    • Example 2: Calculate the hash value of a log based on the value of the db_version field and write the log to a shard of the storage destination that you specify based on the hash value.
      • Raw log:
        __topic__:
db_name: db-01
db_version:5.6

__topic__:
db_name: db-02
db_version:5.7
      • Transformation rule:
        e_output(name="target1", hash_key_field="db_version")
      • Result:
        # For example, the storage destination named target1 has two shards. 
# The MD5 hash range of Shard 0 is [00000000000000000000000000000000,80000000000000000000000000000000). 
# The MD5 hash range of Shard 1 is [80000000000000000000000000000000,ffffffffffffffffffffffffffffffff). 
# The hash values for logs whose values of the db_version field are 5.6 and 5.7 are 0ebe1a34e990772a2bad83ce076e0766 and f1867131d82f2256b4521fe34aec2405. 

# Shard 0:
__topic__:
db_name: db-01
db_version:5.6

# Shard 1:
__topic__:
db_name: db-02
db_version:5.7
    • Example 3: Specify the hash value for a log and write the log to a shard of the storage destination that you specify based on the hash value.
      • Raw log:
        __topic__:
db_name: db-01
db_version:5.6

__topic__:
db_name: db-02
db_version:5.7
      • Transformation rule:
        e_output(name="target1", hash_key="00000000000000000000000000000000")
      • Result:
        # For example, the storage destination has two shards. 
# The MD5 hash range of Shard 0 is [00000000000000000000000000000000,80000000000000000000000000000000). 
# The MD5 hash range of Shard 1 is [80000000000000000000000000000000,ffffffffffffffffffffffffffffffff). 

# Shard 0:
__topic__:
db_name: db-01
db_version:5.6

__topic__:
db_name: db-02
db_version:5.7

# Shard 1:
None

  • References

    This function can be used together with other functions. For more information, see Aggregate data from multiple source Logstores.

e_to_metric

The e_to_metric function converts logs to metrics that can be stored in a Metricstore.

  • Syntax

    e_to_metric(names=None, labels=None, time_field='__time__', time_precision='s', ignore_none_names=True, ignore_none_labels=True)

  • Parameters

    Parameter Type Required Description
    names String, string list, or tuple list Yes The name of the metric. The value of the names parameter can be a single string, string list, or tuple list. The value is the name of a field in the log that you want to convert.
    • String: The name of a log field is used as the name of a metric. The value is a string. Example: rt. A metric record that contains __name__:rt is returned.
    • String list: The names of log fields are used as the names of metrics. The value consists of multiple strings. Example: ["rt", "qps"]. In this example, two metric records are returned. One record contains __name__:rt and the other record contains __name__:qps.
    • Tuple list: The names of multiple log fields are used as the names of metrics, and the metrics are renamed. The value consists of multiple tuples. Example: [("rt","max_rt"),("qps", "total_qps")]. In this example, the first element of a tuple is the name of the raw log field. The second element of a tuple is the name of the metric that is generated after transformation. Two metric records are returned. One record contains __name__:max_rt and the other record contains __name__:total_qps.
    labels String, string list, or tuple list No The label of the metric. The value of the labels parameter can be a single string, string list, or tuple list. The value is the name of a field in the log that you want to convert.
    Note In the following list, host and app are the names of log fields, and hostvalue and appvalue are the values of the log fields.
    • String: The name of a log field is used as the label of a metric. The value is a string. Example: host. A metric record that contains __label__:host#$#hostvalue is returned.
    • String list: The names of log fields are used as the labels of metrics. The value consists of multiple strings. Example: ["host", "app"]. In this example, two metric records are returned. One record contains __label__:host#$#hostvalue and the other record contains __label__:app#$#appvalue.
    • Tuple list: The names of multiple log fields are used as the labels of metrics, and the labels are renamed. The value consists of multiple tuples. Example: [("host","hostname"),("app", "appname")]. In this example, the first element of a tuple is the name of the raw log field. The second element of a tuple is the label of the metric that is generated after transformation. Two metric records are returned. One record contains __label__:hostname#$#hostvalue and the other record contains __label__:appname#$#appvalue.
    time_field String No The time field of the metric. By default, the __time__ field in a log is used as the time field of a metric.
    time_precision Int No The unit of the time field in the raw log. Supported units are seconds, milliseconds, microseconds, and nanoseconds. By default, logs are stored by second. For example, time_field="ms" indicates that the unit of the time field in the raw log is milliseconds.
    ignore_none_names Boolean No Specifies whether to skip the conversion from a log to a metric if the log field does not exist. Valid values:
    • True: skips conversion from a log to a metric. This is the default value.
    • False: does not skip the conversion and reports an error.
    ignore_none_labels Boolean No Specifies whether to skip the conversion from a log to a metric if the log field does not exist. Valid values:
    • True: skips conversion from a log to a metric. This is the default value.
    • False: does not skip the conversion and reports an error.

  • Response

    A metric is returned.

  • Examples

    • Example 1: Convert a log that contains the rt field to a metric.
      • Raw log:
        __time__: 1614739608
rt: 123
      • Transformation rule:
        e_to_metric(names="rt")
      • Result:
        __labels__:
__name__:rt
__time_nano__:1614739608000000000
__value__:123.0
    • Example 2: Convert a log that contains the rt field to a metric and set labels to host.
      • Raw log:
        __time__: 1614739608
rt: 123
host: myhost
      • Transformation rule:
        e_to_metric(names="rt", labels="host")
      • Result:
        __labels__:host#$#myhost
__name__:rt
__time_nano__:1614739608000000000
__value__:123.0
    • Example 3: Convert a log that contains the rt and qps fields to a metric and set labels to host.
      • Raw log:
        __time__: 1614739608
rt: 123
qps: 10
host: myhost
      • Transformation rule:
        e_to_metric(names=["rt","qps"], labels="host")
      • Result:
        __labels__:host#$#myhost
__name__:rt
__time_nano__:1614739608000000000
__value__:123.0

__labels__:host#$#myhost
__name__:qps
__time_nano__:1614739608000000000
__value__:10.0
    • Example 4: Convert a log that contains the rt and qps fields to a metric, rename the fields rt and qps to max_rt and total_qps, and set labels to host.
      • Raw log:
        __time__: 1614739608
rt: 123
qps: 10
host: myhost
      • Transformation rule:
        e_to_metric(names=[("rt","max_rt"),("qps","total_qps")], labels="host")
      • Result:
        __labels__:host#$#myhost
__name__:max_rt
__time_nano__:1614739608000000000
__value__:123.0

__labels__:host#$#myhost
__name__:total_qps
__time_nano__:1614739608000000000
__value__:10.0
    • Example 5: Convert a log that contains the rt and qps fields to a metric, rename the fields rt and qps to max_rt and total_qps, set labels to host, and rename host to hostname.
      • Raw log:
        __time__: 1614739608
rt: 123
qps: 10
host: myhost
      • Transformation rule:
        e_to_metric(names=[("rt","max_rt"),("qps","total_qps")], labels=[("host","hostname")])
      • Result:
        __labels__:hostname#$#myhost
__name__:max_rt
__time_nano__:1614739608000000000
__value__:123.0

__labels__:hostname#$#myhost
__name__:total_qps
__time_nano__:1614739608000000000
__value__:10.0
    • Example 6: Convert a log that contains the remote_user1 and request_length fields to a metric, rename the fields remote_user1 and request_length to remote_user2 and request_length1, and set labels to status1.
      • Raw log:
        __time__:1652943594
remote_user:89
request_length:4264
request_method:GET
status:200
      • Transformation rule:
        # The remote_user1 and status1 fields do not exist in the log and the conversion is skipped. 
e_to_metric(
    names=[("remote_user1", "remote_user2"), ("request_length", "request_length1")],
    labels="status1",
    ignore_none_names=True,
    ignore_none_labels=True,
)
      • Result:
        __labels__:
__name__:request_length1
__time_nano__:1652943594000000000
__value__:4264.0
    • Example 7: Convert a log that contains the remote_user field to a metric, set labels to status, and specify milliseconds as the unit of the time field in the log.
      • Raw log:
        __time__:1652943594
remote_user:89
request_length:4264
request_method:GET
status:200
      • Transformation rule:
        e_to_metric(
    names="remote_user",
    labels="status",
    time_precision="ms",
    ignore_none_names=True,
    ignore_none_labels=True,
)
      • Result:
        __labels__:status#$#200
__name__:remote_user
__time_nano__:1652943594000000
__value__:89.0
    • Example 8: Convert a log that contains the remote_user field to a metric, set labels to status, specify the time field in the log as the time field of the metric, and specify nanoseconds as the unit of the time field in the log.
      • Raw log:
        time:1652943594
remote_user:89
request_length:4264
request_method:GET
status:200
      • Transformation rule:
        e_to_metric(
    names="remote_user",
    labels="status",
    time_field="time",
    time_precision="ns",
    ignore_none_names=True,
    ignore_none_labels=True,
)
      • Result:
        __labels__:status#$#200
__name__:remote_user
__time_nano__:1652943594
__value__:89.0

  • References

    This function can be used together with other functions. For more information, see Convert logs to metrics.