Categories of data transformation functions - Simple Log Service

This topic describes the functions that you can call to transform data in Simple Log Service.

Global processing functions

Category	Function	Description
Flow control functions	e_if	Performs an operation if a specified condition is met. You can specify multiple condition-operation pairs. If a condition is met, the function performs the operation that corresponds to the condition. If the condition is not met, the function does not perform the operation, but evaluates the next condition. If the function performs an operation that deletes a log, the function no longer performs other operations on the log.
	e_if_else	Performs an operation based on the evaluation result of a condition.
	e_switch	Performs an operation if a specified condition is met. You can specify multiple condition-operation pairs. If a condition is met, the function performs the operation that corresponds to the condition and returns the result. If the condition is not met, the function does not perform the operation, but evaluates the next condition. If no specified conditions are met and the default parameter is configured, the function performs the operation that is specified by the default parameter and returns the result. If the function performs an operation that deletes a log, the function no longer performs other operations on the log.
	e_compose	Combines multiple operations. The function is commonly used in the e_if, e_switch, or e_if_else function. The function performs specified operations on a log in sequence and returns the result. If the function performs an operation that deletes a log, the function no longer performs other operations on the log.
Event processing functions	e_drop	Discards a log if a specified condition is met.
	e_keep	Retains a log if a specified condition is met.
	e_split	Splits a log into multiple logs based on the value of a specified field. You can also use the JMESPath expression to extract the value of the field, and then split the log.
	e_output	Writes a log to a specified Logstore. You can specify the topic, source, tags and MD5 hash keys of a shard for the log. The log is deleted after it is written to the specified Logstore. The system no longer transforms the log.
	e_coutput	Writes a log to a specified Logstore. You can specify the topic, source, tags and MD5 hash keys of a shard for the log. The log is retained after it is written to the specified Logstore. The system continues to transform the retained log.
	e_to_metric	Converts logs to metrics that can be stored in a Metricstore.
Field processing functions	v	Extracts the value of a field from a log. If you specify the names of multiple fields for the function, the function returns the value of the first field that exists in the log.
	e_set	Adds a field or specifies a new value for an existing field.
	e_drop_fields	Deletes the log fields that meet a specified condition.
	e_keep_fields	Retains the log fields that meet a specified condition.
	e_pack_fields	Encapsulates log fields and assigns the log fields as a value to a new field.
	e_rename	Renames the log fields that meet a specified condition.
Value extraction functions	e_regex	Extracts the value of a field based on a regular expression and assigns the value to other fields.
	e_json	Manages JSON objects in a specified field in a log. You can configure the parameters to expand JSON data, extract JSON data by using the JMESPath expression, or expand the extracted JSON data.
	e_kv	Extracts key-value pairs from multiple input fields by using a specified quote.
	e_kv_delimit	Extracts key-value pairs from input fields by using a specified delimiter.
	e_csv	Extracts multiple fields from a specified field by using a specified delimiter and predefined field names. The default delimiter is a comma (,).
	e_tsv	Extracts multiple fields from a specified field by using a specified delimiter and predefined field names. The default delimiter is `\t`.
	e_psv	Extracts multiple fields from a specified source field by using a specified delimiter and predefined field names. The default delimiter is a vertical bar (\|).
	e_syslogrfc	Calculates the values of the facility and severity fields and returns the value of the facilitylabel field that indicates level information. The function calculates the values based on the value of the priority field and the specified syslog protocol.
	e_anchor	Extracts strings by using the rules specified by anchor_rules.
Mapping and enrichment functions	e_dict_map	Maps the value of an input field to a value in a specified dictionary and returns a new field.
	e_table_map	Maps the value of an input field to a row in a specified table and returns a new field.
	e_tablestore_map	Enriches a raw log by using a data table in Tablestore as the dimension table.
	e_redis_map	Enriches a raw log by using a data table in ApsaraDB for Redis as the dimension table.

Expression functions

Category	Function	Description
Event check functions	e_has	Checks whether a field exists.
	e_not_has	Checks whether a field does not exist.
	e_search	Searches for a log by using a query syntax that is similar to Lucene.
	e_match, e_match_all, and e_match_any	Checks whether the value of a log field meets the conditions specified in a regular expression.
Operator functions	op_if	Returns the value of an expression based on a specified condition.
	op_ifnull	Returns the value of the first expression whose value is not None.
	op_coalesce	Returns the value of the first expression whose value is not None.
	op_nullif	Returns none if the value of Expression 1 is equal to the value of Expression 2. If the values of Expression 1 and Expression 2 are different, the value of Expression 1 is returned.
	op_and	Evaluates the specified expressions by using the logical AND operator and returns True if all specified expressions evaluate to true. The value of each expression can be of an arbitrary data type.
	op_not	Evaluates a specified expression by using the logical NOT operator and returns the inverse Boolean value of the specified expression. The value of the expression can be of an arbitrary data type.
	op_or	Evaluates the specified expressions by using the logical OR operator, and returns True if a specified expression evaluates to true or returns False if all specified expressions evaluate to false. The value of each expression can be of an arbitrary data type.
	op_eq	Returns True or False based on the `a==b` condition.
	op_ge	Returns True or False based on the `a>=b` condition.
	op_gt	Returns True or False based on the `a>b` condition.
	op_le	Returns True or False based on the `a<=b` condition.
	op_lt	Returns True or False based on the `a<b` condition.
	op_ne	Returns True or False based on the `a!=b` condition.
	op_len	Calculates the number of characters in a text string. This function applies to strings or expressions that return tuples, lists, or dictionaries.
	op_in	Checks whether a string, tuple, list, or dictionary contains a specified element and returns True or False.
	op_not_in	Checks whether a string, tuple, list, or dictionary does not contain a specified element and returns True or False.
	op_slice	Extracts strings from a specified string, array, or tuple.
	op_index	Returns the element that corresponds to the index of a specified string, array, or tuple.
	op_add	Calculates the sum of multiple values. The values can be strings or numbers.
	op_max	Returns the largest value among the values of multiple fields or expressions.
	op_min	Returns the smallest value among the values of multiple fields or expressions.
Conversion functions	ct_int	Converts the value of a field or an expression to an integer.
	ct_float	Converts the value of a field or an expression to a floating-point number.
	ct_str	Converts the value of a field or an expression to a string.
	ct_bool	Converts the value of a field or an expression to a Boolean value.
	ct_chr	Converts the ANSI or Unicode value of a field or an expression to a character.
	ct_ord	Converts the value of a field or an expression to an ANSI value or a Unicode value.
	ct_hex	Converts the value of a field or an expression to a hexadecimal number.
	ct_oct	Converts the value of a field or an expression to an octal number.
	ct_bin	Converts the value of a field or an expression to a binary number.
	bin2oct	Converts a binary byte string to an octal string.
	bin2hex	Converts a binary byte string to a hexadecimal string.
Arithmetic functions	op_abs	Returns the absolute value of an input value.
	op_div_floor	Returns the integer part of the quotient of two input values.
	op_div_true	Returns the quotient of two input values.
	op_pow	Returns an input value raised to a specified power.
	op_mul	Returns the product of two input values.
	op_neg	Returns the opposite number of an input value.
	op_mod	Returns the remainder of an input value divided by the other input value.
	op_sub	Returns the difference between two input values.
	op_round	Returns an input value rounded.
	op_sum	Returns the sum of input values.
	mat_ceil	Rounds an input value rounded up to the nearest integer.
	mat_exp	Returns Euler's number raised to the power of an input value.
	mat_fabs	Returns the absolute value of an input value.
	mat_floor	Rounds an input value down to the nearest integer.
	mat_log	Returns the logarithm of an input value with the base specified by the other input value.
	mat_log10	Returns the base-10 logarithm of an input value.
	mat_sqrt	Returns the square root of an input value.
	mat_degrees	Converts radians to degrees.
	mat_radians	Converts degrees to radians.
	mat_sin	Returns the sine of an input value in radians.
	mat_cos	Returns the cosine of an input value in radians.
	mat_tan	Returns the tangent of an input value in radians.
	mat_acos	Returns the arc cosine of an input value in radians.
	mat_asin	Returns the arc sine of an input value in radians.
	mat_atan	Returns the arc tangent of an input value in radians.
	mat_atan2	Returns the arc tangent of X and Y coordinates.
	mat_atanh	Returns the inverse hyperbolic tangent of an input value.
	mat_hypot	Returns the Euclidean norm of two input values.
String functions	str_format	Formats strings.
	str_join	Concatenates input strings to generate a new string by using a specified connector.
	str_zip	Concurrently splits two values or strings that are returned by expressions and combines the results into one string.
	str_encode	Encodes a string by using a specified encoding format.
	str_decode	Decodes an input value by using a specified encoding format.
	str_hex_escape_encode	Escapes special characters. The function can escape hexadecimal characters to Chinese characters.
	str_sort	Sorts a specified object.
	str_reverse	Reverses a string.
	str_replace	Replaces an existing string with a specified string based on a specified rule.
	str_logtash_config_normalize	Converts data in the Logstash configuration language to the JSON format.
	str_translate	Replaces specified characters in a string with mapping characters.
	str_strip	Deletes specified characters from a string.
	str_lstrip	Deletes specified characters from the start of a string.
	str_rstrip	Deletes specified characters from the end of a string.
	str_lower	Converts all uppercase letters in a string to lowercase letters.
	str_upper	Converts all lowercase letters in a string to uppercase letters.
	str_title	Capitalizes the first letter of each word in a string and converts the other letters in the string to lowercase letters.
	str_capitalize	Capitalizes the first letter of a string and converts the other letters in the string to lowercase letters.
	str_swapcase	Converts the uppercase letters in a string to lowercase letters and lowercase letters to uppercase letters.
	str_count	Counts the number of occurrences of a character in a string.
	str_find	Checks whether a string contains a specified substring.
	str_rfind	Returns the position of the last occurrence of a specified character in a string.
	str_endswith	Checks whether a string ends with a specified suffix.
	str_startswith	Checks whether a string starts with a specified string.
	str_split	Splits a string by using a specified delimiter.
	str_splitlines	Splits a string by using a line feed.
	str_partition	Splits a string into three parts from left to right by using a specified delimiter.
	str_rpartition	Splits a string into three parts from right to left by using a specified delimiter.
	str_center	Pads a string to a specified length by using a specified character.
	str_ljust	Pads a string to a specified length by using a specified character from the end of the string.
	str_rjust	Pads a string to a specified length by using a specified character from the start of the string.
	str_zfill	Pads a string to a specified length by using 0 from the start of the string.
	str_expandtabs	Converts `\t` in a string to spaces.
	str_isalnum	Checks whether a string contains only letters and digits.
	str_isalpha	Checks whether a string contains only letters.
	str_isascii	Checks whether a string is in the ASCII table.
	str_isdecimal	Checks whether a string contains only decimal characters.
	str_isdigit	Checks whether a string contains only digits.
	str_isidentifier	Checks whether a string is a valid Python identifier or checks whether a variable name is valid.
	str_islower	Checks whether a string contains lowercase letters.
	str_isnumeric	Checks whether a string contains digits.
	str_isprintable	Checks whether all characters in a string are printable characters.
	str_isspace	Checks whether a string contains only spaces.
	str_istitle	Checks whether the first letter of each word in a string is in uppercase and the other letters in the string are in lowercase.
	str_isupper	Checks whether all letters in a string are in uppercase.
	str_uuid	Generates a random universally unique identifier (UUID).
Date and time functions	dt_parse	Converts a value or the value of a time expression to a datetime object.
	dt_str	Converts a value or the value of a time expression to a string.
	dt_parsetimestamp	Converts a value or the value of a time expression to a UNIX timestamp.
	dt_prop	Returns a specific attribute of a value, or returns a specific attribute of the value of a time expression. The attribute can be day or year.
	dt_now	Returns the current date and time.
	dt_today	Return only the current date.
	dt_utcnow	Returns the current datetime object in the current time zone.
	dt_fromtimestamp	Converts a UNIX timestamp to a datetime object.
	dt_utcfromtimestamp	Converts a UNIX timestamp to a datetime object in the current time zone.
	dt_strptime	Parses a time string into a datetime object.
	dt_currentstamp	Returns the current UNIX timestamp.
	dt_totimestamp	Converts a datetime object to a UNIX timestamp.
	dt_strftime	Converts a datetime object to a string in a specified format.
	dt_strftimestamp	Converts a UNIX timestamp to a string in a specified format.
	dt_truncate	Extracts a time value from a value or the value of a time expression based on a specified time granularity.
	dt_add	Changes a value or the value of a time expression based on a specified time granularity.
	dt_MO	Offsets a specified time to the date of the previous or following Nth Monday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_TU	Offsets a specified time to the date of the previous or following Nth Tuesday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_WE	Offsets a specified time to the date of the previous or following Nth Wednesday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_TH	Offsets a specified time to the date of the previous or following Nth Thursday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_FR	Offsets a specified time to the date of the previous or following Nth Friday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_SA	Offsets a specified time to the date of the previous or following Nth Saturday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_SU	Offsets a specified time to the date of the previous or following Nth Sunday. The offset value N is passed to the `weekday` parameter of the `dt_add` function.
	dt_astimezone	Converts a value or the value of a time expression to a datetime object in a specified time zone.
	dt_diff	Returns the difference between two values or between the values of two time expressions based on a specified time granularity.
Regular expression functions	regex_select	Extracts a value that matches a regular expression.
	regex_findall	Extracts all values that match a regular expression.
	regex_match	Checks whether a value matches a regular expression.
	regex_replace	Replaces the characters that match a regular expression in a string.
	regex_split	Splits a string into an array of strings.
Grok function	grok	Extracts a value that matches a regular expression.
Structured data functions	json_select	Extracts or calculates specific values from a JSON expression by using JMESPath.
	json_parse	Parses a value into a JSON object.
	xml_to_json	Converts XML data to JSON data, and then expands the converted data.
IP address parsing functions	geo_parse	Identifies the city, province, and country based on an IP address.
	ip_cidrmatch	Checks whether an IP address belongs to a Classless Inter-Domain Routing (CIDR) block.
	ip_version	Checks whether the version of an IP address is IPv4 or IPv6.
	ip_type	Identifies the type of an IP address and checks whether the type of the IP address is private or public.
	ip_makenet	Converts an IP address to a CIDR block.
	ip_to_format	Converts the format of a CIDR block to a format that specifies the netmask or prefix length of the CIDR block.
	ip_overlaps	Checks whether two CIDR blocks overlap.
	ip2long	Converts an IP address to a value of the long type.
	long2ip	Converts a value of the long type to an IP address.
Encoding and decoding functions	url_encoding	Performs URL encoding on data.
	url_decoding	Performs URL decoding on data.
	protobuf_decoding	Parses data into the JSON format by using a specified Protobuf template.
	str_encode	Encodes a string by using a specified encoding format.
	str_decode	Decodes an input value by using a specified encoding format.
	base64_encoding	Encodes data by using the Base64 algorithm.
	base64_decoding	Decodes data by using the Base64 algorithm.
	html_encoding	Encodes data in the HTML format.
	html_decoding	Decodes HTML-encoded data.
	md5_encoding	Encodes data by using the MD5 algorithm.
	sha1_encoding	Encodes data by using the SHA1 algorithm.
	crc32_encoding	Calculates a cyclic redundancy check (CRC) code for data.
	gzip_compress	Compresses and encodes data.
	gzip_decompress	Decompresses compressed data.
	zlib_compress	Compresses and encodes data.
	zlib_decompress	Decompresses compressed data.
	aes_encrypt	Encrypts data by using the AES algorithm.
	aes_decrypt	Decrypts data by using the AES algorithm.
	jwt_encoding	Encodes JSON data based on the JSON Web Token (JWT) standard.
	jwt_decoding	Decodes data to raw JSON data based on the JWT standard.
	hashids_encoding	Encodes data by using the Hashids library.
	hashids_decoding	Decodes data that is encoded by using the Hashids library.
Parsing functions	ua_parse_device	Parses User-Agent and returns the device information.
	ua_parse_os	Parses User-Agent and returns the operating system information.
	ua_parse_agent	Parses User-Agent and returns the browser information.
	ua_parse_all	Parses User-Agent and returns all information.
	url_parse	Parses a URL and returns the components of the URL.
	url_parse_qs	Parses the query string of a URL and returns the components of the query string.
List functions	lst_make	Constructs a list.
	lst_insert	Inserts elements to a specified position in a list.
	lst_append	Appends elements to a list.
	lst_delete_at	Deletes the element at a specified position from a list.
	lst_reverse	Reverses the order of elements in a list.
	lst_get	Returns the element at a specified position in a list or a tuple.
Dictionary functions	dct_make	Constructs a dictionary.
	dct_update	Updates a dictionary.
	dct_delete	Deletes key-value pairs from a dictionary.
	dct_keys	Returns the keys of a dictionary.
	dct_values	Returns the values of a dictionary.
	dct_get	Returns the value that corresponds to a specified key in a dictionary.
Table functions	tab_parse_csv	Constructs a table from comma-separated values (CSV) text.
Table functions	tab_to_dict	Constructs a dictionary from a table.
Resource functions	res_local	Pulls the values of advanced parameters from the current data transformation job.
	res_rds_mysql	Pulls data from a specified table in a database that is created on an ApsaraDB RDS for MySQL instance or obtains the execution result of an SQL statement. The data and result can be updated at regular intervals.
	res_log_logstore_pull	Pulls data from another Logstore when you transform data in a Logstore. You can pull data in a continuous manner.
	res_oss_file	Pulls data from an object in a specified Object Storage Service (OSS) bucket. The data can be updated at regular intervals.