Parsing functions - Simple Log Service - Alibaba Cloud Documentation Center

Functions


Function	Description
ua_parse_device	Parses User-Agent and returns the device information.
ua_parse_os	Parses User-Agent and returns the operating system information.
ua_parse_agent	Parses User-Agent and returns the browser information.
ua_parse_all	Parses User-Agent and returns all information.
url_parse	Parses a URL and returns the components of the URL.
url_parse_qs	Parses the query string of a URL and returns the components of the query string.

Note The User-Agent parsing functions delete fields whose parsed values are None. For example, if the parsed device information is {'brand': None, 'family': 'Other', 'model': None}, the brand and model fields are deleted, and the final parsing result is {'family': 'Other'}.

ua_parse_device

The ua_parse_device function parses User-Agent and returns the device information.

Syntax
```
ua_parse_device(value)
```

Parameters


Parameter	Type	Required	Description
value	String	Yes	The User-Agent string that you want to parse.

Response

A JSON-formatted data set is returned.

Examples

Raw log

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36

Transformation rule

e_set("new_column",ua_parse_device(v("http_user_agent")))

Result

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
new_column:{'family': 'Other'}

ua_parse_os

The ua_parse_os function parses User-Agent and returns the operating system information.

Syntax
```
ua_parse_os(value)
```

Parameters


Parameter	Type	Required	Description
value	String	Yes	The User-Agent string that you want to parse.

Response

A JSON-formatted data set is returned.

Examples

Raw log

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36

Transformation rule

e_set("new_column",ua_parse_os(v("http_user_agent")))

Result

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
new_column:{'family': 'Mac OS X',
                    'major': '10',
                    'minor': '9',
                    'patch': '4'}

ua_parse_agent

The ua_parse_agent function parses User-Agent and returns the browser information.

Syntax
```
ua_parse_agent(value)
```

Parameters


Parameter	Type	Required	Description
value	String	Yes	The User-Agent string that you want to parse.

Response

A JSON-formatted data set is returned.

Examples

Raw log

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36

Transformation rule

e_set("new_column",ua_parse_agent(v("http_user_agent")))

Result

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
new_column:{'family': 'Chrome', 'major': '192', 'minor': '168', 'patch': '0'}

ua_parse_all

The ua_parse_all function parses User-Agent and returns all information.

Syntax
```
ua_parse_all(value)
```

Parameters


Parameter	Type	Required	Description
value	String	Yes	The User-Agent string that you want to parse.

Response

A JSON-formatted data set is returned.

Examples

Raw log

http_user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36

Transformation rule

e_set("new_column",ua_parse_all(v("http_user_agent")))

Result

http_user_agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/192.168.0.0 Safari/537.36
new_column: {
  "user_agent": {
    "family": "Chrome",
    "major": "192",
    "minor": "168",
    "patch": "0"
  },
  "os": {
    "family": "Mac OS X",
    "major": "10",
    "minor": "9",
    "patch": "4"
  },
  "device": {
    "family": "Mac",
    "brand": "Apple",
    "model": "Mac"
  }
}

url_parse

The url_parse function parses a URL and returns the components of the URL.

Syntax

url_parse(url, scheme="", allow_fragments=True)

Parameters


Parameter	Type	Required	Description
value	String	Yes	The URL that you want to parse.
scheme	String	No	The network protocol. This parameter is empty by default. If the URL that you want to parse does not contain a network protocol, the value of this parameter is returned for the scheme parameter in the result.
allow_fragments	Boolean	No	Specifies whether to parse the fragment part in the URL. Valid values: True: parses the fragment part in the URL. A specific value is returned for the fragment parameter. This is the default value. False: does not parse the fragment part in the URL. An empty string is returned for the fragment parameter.

Response

JSON-formatted data is returned. The following table describes the parameters in the result.


Parameter	Description
scheme	The network protocol.
netloc	The network location.
path	The hierarchical path identifier.
query	The query component.
fragment	The fragment identifier.

Examples

Example 1: Use default parameter settings. The components of a URL are returned.

Raw log

content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib

Transformation rule
```
e_set("url",url_parse(v("content")))
```

Result

content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
url:{
        "scheme": "https",
        "netloc": "username:username@example.com:8083",
        "path": "/hello/asdah/;type=docx",
        "query": "filename=python3.docx",
        "fragment": "urllib",
    }

Example 2: Set allow_fragments to False. An empty string is returned for the fragment parameter.

Raw log

content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib

Transformation rule

e_set("url",url_parse(v("content"),allow_fragments=False))

Result

content:https://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
url:{
                "scheme": "https",
        "netloc": "username:username@example.com:8083",
        "path": "/hello/asdah/;type=docx",
        "query": "filename=python3.docx",
        "fragment": "",
    }

Example 3: Set scheme to https and set allow_fragments to False. https is returned for the scheme parameter, and an empty string is returned for the fragment parameter.

Raw log

content://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib

Transformation rule

e_set("url",url_parse(v("content"),scheme="https", allow_fragments=False))

Result

content://username:username@example.com:8083/hello/asdah/;type=docx?filename=python3.docx#urllib
url:{
       "scheme": "https",
        "netloc": "username:username@example.com:8083",
        "path": "/hello/asdah/;type=docx",
        "query": "filename=python3.docx",
        "fragment": "",
    }

url_parse_qs

The url_parse_qs function parses the query string of a URL and returns the components of the query string.

Syntax

url_parse_qs(
    url_qs,
    keep_blank_values=False,
    strict_parsing=False,
    encoding="utf-8",
    errors="replace",
    ignore_multi_fields=True,
)

Parameters


Parameter	Type	Required	Description
url_qs	String	Yes	The URL query string that you want to parse.
keep_blank_values	Boolean	No	Specifies whether to return parameters that are empty. Valid values: False: does not return parameters that are empty. This is the default value. True: returns parameters that are empty and returns empty strings as values for the parameters.
strict_parsing	Boolean	No	Specifies whether to handle a parsing error. Valid values: True: triggers a ValueError exception for the parsing error. False: ignores the parsing error. This is the default value.
encoding	String	No	The encoding method. Default value: utf-8. Escape characters that contain percent signs (%) are parsed into Unicode characters. ASCII is supported.
errors	String	No	The method to process characters that cannot be recognized based on the encoding method. Valid values: ignore: ignores the characters. strict: reports an error and discards the log that contains the characters. replace: replaces the characters with question marks (?). This is the default value. xmlcharrefreplace: replaces the characters with XML characters.
ignore_multi_fields	Num	No	The number of values that are returned for a single parameter. Valid values: True: returns only the first value for each parameter. The return value is a string. This is the default value. False: returns all values for each parameter. The return value is a list.

Response

JSON-formatted data is returned. The following table describes the parameters in the result.


Parameter	Description
logType	The type of the log.
uid	The unique identifier of the log.
time	The time of the log.
msg	The information in the log.

Examples

Example 1: Set keep_blank_values to True. Parameters that are empty are returned.

Raw log

content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp

Transformation rule

e_set("url",url_parse_qs(v("content"), keep_blank_values=True))

Result

content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
url:{
        "logType": "net_wheel_log",
        "uid": "62452****",
        "vid": "6.1.0_gf_pc",
        "asb": "1206427",
        "git": "",
        "time": "22-11-3 11:49:33",
        "operatingSystem": "Windows 10  (10.0.0) 64bit",
        "deviceModel": "System Product Name (System manufacturer)",
        "graphicsDeviceName": "NVIDIA GeForce GTX 1650",
        "graphicsDeviceType": "Direct3D11",
        "graphicsDeviceVendor": "NVIDIA",
        "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]",
        "graphicsMemorySize": "3962",
        "systemMemorySize": "8127",
        "processorCount": "6",
        "processorFrequency": "3000",
        "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz",
        "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****",
        "guessUID": "62452****",
        "networkReachability": "wifi",
        "msg": "GetAuthkeyRsp",
    }

Example 2: Set keep_blank_values to False. Parameters that are not empty are returned.

Raw log

content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp

Transformation rule
```
e_set("url",url_parse_qs(v("content")))
```

Result

content:logType=net_wheel_log&uid=62452****&vid=6.1.0_gf_pc&asb=1206427&git=&time=22-11-3+%e4%b8%8a11%e6%97%b649%e5%88%8633%e7%a7%92&operatingSystem=Windows+10++(10.0.0)+64bit&deviceModel=System+Product+Name+(System+manufacturer)&graphicsDeviceName=NVIDIA+GeForce+GTX+1650&graphicsDeviceType=Direct3D11&graphicsDeviceVendor=NVIDIA&graphicsDeviceVersion=Direct3D+11.0+%5blevel+11.1%5d&graphicsMemorySize=3962&systemMemorySize=8127&processorCount=6&processorFrequency=3000&processorType=Intel(R)+Core(TM)+i5-9500F+CPU+%40+3.00GHz&deviceID=96da5902a042a5f84118995f88373f73650e76be166589726****&guessUID=62452****&networkReachability=wifi&msg=GetAuthkeyRsp
url:{
        "logType": "net_wheel_log",
        "uid": "62452****",
        "vid": "6.1.0_gf_pc",
        "asb": "1206427",
        "time": "22-11-3 11:49:33",
        "operatingSystem": "Windows 10  (10.0.0) 64bit",
        "deviceModel": "System Product Name (System manufacturer)",
        "graphicsDeviceName": "NVIDIA GeForce GTX 1650",
        "graphicsDeviceType": "Direct3D11",
        "graphicsDeviceVendor": "NVIDIA",
        "graphicsDeviceVersion": "Direct3D 11.0 [level 11.1]",
        "graphicsMemorySize": "3962",
        "systemMemorySize": "8127",
        "processorCount": "6",
        "processorFrequency": "3000",
        "processorType": "Intel(R) Core(TM) i5-9500F CPU @ 3.00GHz",
        "deviceID": "96da5902a042a5f84118995f88373f73650e76be166589726****",
        "guessUID": "62452****",
        "networkReachability": "wifi",
        "msg": "GetAuthkeyRsp",
    }

Example 3: Set ignore_multi_fields to True. Only the first value for each parameter is returned.

Raw log

content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456

Transformation rule
```
e_set("url",url_parse_qs(v("content")))
```

Result

content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
url:{
    "logType": "net_log",
    "uid": "62452****",
    "x": "1",
    "asb": "123"
}

Example 4: Set ignore_multi_fields to False. All values for each parameter are returned.

Raw log

content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456

Transformation rule

e_set("url",url_parse_qs(v("content"),ignore_multi_fields=False))

Result

content:logType=net_log&uid=62452****&x=1&x=2&x=3&asb=123&asb=456
url:{
    "logType": ["net_log"],
      "uid": ["62452****"],
      "x": ["1", "2", "3"],
      "asb": ["123", "456"],
}