Replace strings

0.0.201

If you use Logtail to collect logs, you can use the processor_string_replace plug-in to replace specific strings by specifying a string or a regular expression, or removing escape characters. This topic describes the parameters that are used to configure the processor_string_replace plug-in and provides configuration examples.

Important

Form configuration: Available when collecting text logs and container stdout and stderr.
Editor configuration in JSON: Not available when collecting text logs.

Entry point

If you want to use a Logtail plug-in to process logs, you can add a Logtail plug-in configuration when you create or modify a Logtail configuration. For more information, see Overview of Logtail plug-ins for data processing.

Plug-in configuration description

Important

Logtail V1.6.0 or later supports the processor_string_replace plug-in.

Form configuration

Editor configuration in JSON

Set the Processor Type parameter to String Replacement. Then, configure other parameters based on the following table.

Parameter	Description
Original Field	The name of the original field.
Match Mode	The method that is used to replace the content. Valid values: String Match: replaces the content that matches a specified string with a different string. Regex Match: replaces the content that matches a specified regular expression with a string. Remove Escape Character: removes escape characters.
Matched Content	The content that you want to replace. If you set the Match Mode parameter to String Match, specify a string that you want to replace. If multiple strings are matched, all matched strings are replaced. If you set the Match Mode parameter to Regex Match, specify a regular expression that you want to use to match specific content. If multiple strings are matched, all matched strings are replaced. You can also use the group method in a regular expression to match specified groups. If you set the Match Mode parameter to Remove Escape Character, you do not need to configure this parameter.
Replaced By	The string that is used to replace the matched content. If you set the Match Mode parameter to String Match, specify a string that you want to use to replace the matched content. If you set the Match Mode parameter to Regex Match, specify a string that you want to use to replace the matched content. You can also use the group method in a regular expression. If you set the Match Mode parameter to Remove Escape Character, you do not need to configure this parameter.
New Field	The new field that is assigned to the string used to replace the matched content.

Set the type parameter to processor_string_replace. Then, configure other parameters in detail based on the following table.

Parameter	Type	Required	Description
`SourceKey`	String	Yes	The name of the original field.
`Method`	String	Yes	The method that is used to replace the content. Valid values: `const`: replaces the content that matches a specified string with a different string. `regex`: replaces the content that matches a specified regular expression with a string. `unquote`: removes escape characters.
`Match`	String	No	The content that you want to replace. If you set the `Method` parameter to `const`, specify a string that you want to replace. If multiple strings are matched, all matched strings are replaced. If you set the `Method` parameter to `regex`, specify a regular expression that you want to use to match specific content. If multiple strings are matched, all matched strings are replaced. You can also use the group method in a regular expression to match specified groups. If you set the `Method` parameter to `unquote`, you do not need to configure this parameter.
`ReplaceString`	String	No	The string that is used to replace the matched content. Default value: "". If you set the `Method` parameter to `const`, specify a string that you want to use to replace the matched content. If you set the `Method` parameter to `regex`, specify a string that you want to use to replace the matched content. You can also use the group method in a regular expression. If you set the `Method` parameter to `unquote`, you do not need to configure this parameter.
`DestKey`	String	No	The new field that is assigned to the string used to replace the matched content. By default, no new fields are added.

Configuration examples

Replace the content that matches a specified string

Replace Error: in the value of the content field with an empty string.

Form configuration

Editor configuration in JSON

Raw log:

"content": "2023-05-20 10:01:23 Error: Unable to connect to database."

Logtail plug-in configuration for data processing:

Result:

"content": "2023-05-20 10:01:23 Unable to connect to database."

Raw log:

"content": "2023-05-20 10:01:23 Error: Unable to connect to database."

Logtail plug-in configuration for data processing:

{
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "const",
        "Match": "Error: ", 
        "ReplaceString": ""
      }
    }
  ]
}

Result:

"content": "2023-05-20 10:01:23 Unable to connect to database.",

Replace the content that matches a specified regular expression

Replace the string that matches the \\u\w+\[\d{1,3};*\d{1,3}m|N/A regular expression in the value of the content field with an empty string.

Form configuration

Editor configuration in JSON

Raw log:

"content": "2022-09-16 09:03:31.013 \u001b[32mINFO \u001b[0;39m \u001b[34m[TID: N/A]\u001b[0;39m [\u001b[35mThread-30\u001b[0;39m] \u001b[36mc.s.govern.polygonsync.job.BlockTask\u001b[0;39m : Block collection------End------\r"

Logtail plug-in configuration for data processing:

Result:

"content": "2022-09-16 09:03:31.013 INFO [TID: ] [Thread-30] c.s.govern.polygonsync.job.BlockTask : Block collection------End------\r",

Raw log:

"content": "2022-09-16 09:03:31.013 \u001b[32mINFO \u001b[0;39m \u001b[34m[TID: N/A]\u001b[0;39m [\u001b[35mThread-30\u001b[0;39m] \u001b[36mc.s.govern.polygonsync.job.BlockTask\u001b[0;39m : Block collection------End------\r"

Logtail plug-in configuration for data processing:

{
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "regex",
        "Match": "\\\\u\\w+\\[\\d{1,3};*\\d{1,3}m|N/A", 
        "ReplaceString": ""
      }
    }
  ]
}

Result:

"content": "2022-09-16 09:03:31.013 INFO [TID: ] [Thread-30] c.s.govern.polygonsync.job.BlockTask : Block collection------End------\r",

Use the group method in a regular expression to match specified groups

Use the group method in a regular expression to replace 16 in the value of the content field with */24 and specify the new_ip field to store */24.

Important

If you use the group method in a regular expression, the string that is used to replace the matched content cannot contain braces ({}). You can specify a string in the $1 or $2 format.

Form configuration

Editor configuration in JSON

Raw log:
```
"content": "10.10.239.16"
```
Logtail plug-in configuration for data processing:

Result:

"content": "10.10.239.16",
"new_ip": "10.10.239.*/24",

Raw log:
```
"content": "10.10.239.16"
```

Logtail plug-in configuration for data processing:

{
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "regex",
        "Match": "(\\d.*\\.)\\d+", 
        "ReplaceString": "$1*/24",
        "DestKey": "new_ip"
      }
    }
  ]
}

Result:

"content": "10.10.239.16",
"new_ip": "10.10.239.*/24",

Remove escape characters

Form configuration

Editor configuration in JSON

Raw log:

"content": "{\\x22UNAME\\x22:\\x22\\x22,\\x22GID\\x22:\\x22\\x22,\\x22PAID\\x22:\\x22\\x22,\\x22UUID\\x22:\\x22\\x22,\\x22STARTTIME\\x22:\\x22\\x22,\\x22ENDTIME\\x22:\\x22\\x22,\\x22UID\\x22:\\x222154212790\\x22,\\x22page_num\\x22:1,\\x22page_size\\x22:10}"

Logtail plug-in configuration for data processing:

Result:

"content": "{\"UNAME\":\"\",\"GID\":\"\",\"PAID\":\"\",\"UUID\":\"\",\"STARTTIME\":\"\",\"ENDTIME\":\"\",\"UID\":\"2154212790\",\"page_num\":1,\"page_size\":10}",

Raw log:

"content": "{\\x22UNAME\\x22:\\x22\\x22,\\x22GID\\x22:\\x22\\x22,\\x22PAID\\x22:\\x22\\x22,\\x22UUID\\x22:\\x22\\x22,\\x22STARTTIME\\x22:\\x22\\x22,\\x22ENDTIME\\x22:\\x22\\x22,\\x22UID\\x22:\\x222154212790\\x22,\\x22page_num\\x22:1,\\x22page_size\\x22:10}"

Logtail plug-in configuration for data processing:

{
  "processors":[
    {
      "type":"processor_string_replace",
      "detail": {
        "SourceKey": "content",
        "Method": "unquote"
      }
    }
  ]
}

Result:

"content": "{\"UNAME\":\"\",\"GID\":\"\",\"PAID\":\"\",\"UUID\":\"\",\"STARTTIME\":\"\",\"ENDTIME\":\"\",\"UID\":\"2154212790\",\"page_num\":1,\"page_size\":10}",

Feedback