Use Function Compute to perform message cleansing - Function Compute

Common message processing templates, such as the message splitting, dynamic routing, and message enrichment templates, are provided to perform message cleansing. You can directly use a template to process messages or modify the code of the template based on your business requirements. This topic describes the types of templates used to perform message cleansing in ApsaraMQ for RocketMQ and how to use the templates.

Background information

Message cleansing tasks provide basic operator capabilities, and Function Compute provides the underlying logic. After you create a message cleansing task in ApsaraMQ for RocketMQ, you can log on to the Function Compute console to customize the code and modify function configurations.

Operator	Description
Message filtering	Matches the message content based on regular expressions and sends the matched messages to specified destinations. For more information, see Event patterns.
Message conversion	Converts message content based on string match conditions and sends the converted message to specified destinations. For example, character case conversion can be performed. For more information, see Event transformation.
Content segmentation	Splits message content based on regular expressions and sends the split messages to specified destinations.
Dynamic routes	Matches the message content based on regular expressions and sends the matched messages to specified destinations and the unmatched messages to default destinations.
Content enrichment	Enriches message content based on enrichment sources. If the original content of a message contains AccountID, AccountID is used to query the database and obtain the customer region. The system inserts the customer region into the source message body and sends the message body to specified destination services.
Content mapping	Maps message content based on regular expressions. For example, the system can mask sensitive fields in a message or reduce the message size to the minimum size.

Content splitting

Examples

The following message contains a list of students.

message:
[Jack, Male, 17, Class 4; Alice, Female, 17, Class 3; John, Male, 17, Class 4]

You need to split the message into the following messages, each of which contains information about a single student, before you send the messages to specified destination services. Example:

message:
    [Jack, Male, 17, Class 4]
message:
    [Alice, Female, 17, Class 3]
message:
    [John, Male, Class 17, Class 4]

dataclean_split

Procedure

Log on to the ApsaraMQ for RocketMQ console.
In the left-side navigation pane, choose Message Integration > Message Outflow. In the top navigation bar, select a region.
On the Message Outflow page, click Create Task.

In the Create Message Outflow Task panel, configure the following parameters and click Confirm.

Configure the following parameters as described. Retain the default values for other parameters.

Basic Information

Parameter	Description
Task Name.	Enter a name for the task.
Message Outflow Task Type	In this example, select ApsaraMQ for RocketMQ. The following message services are supported: ApsaraMQ for RocketMQ, ApsaraMQ for RabbitMQ, Simple Message Queue (formerly MNS), and ApsaraMQ for Kafka.

Resource Configuration

Parameter	Description
Source
Region	In this example, China (Hangzhou) is selected.
Version	Select the version of ApsaraMQ for RocketMQ. In this example, RocketMQ 5.x is selected.
Instance	The ApsaraMQ for RocketMQ instance that produces messages.
Topic	Select the topic of the source instance.
Target
Version	Select the version of ApsaraMQ for RocketMQ that receives messages. In this example, RocketMQ 5.x is selected.
Instance ID	The ID of the ApsaraMQ for RocketMQ instance that receives messages.
Topic	The topic that you want to use to receive the messages.

Data Processing
- Message Filtering: Select None.
- Message Conversion: Select Custom Configuration. Set Message Body to Data Cleansing and select Create Function Template. Set Function Template to Content Splitting transform_split and modify the function code based on your business requirements.

After the creation, you can log on to the Function Compute console to view the created service and function,

Dynamic routing

Examples

The following message contains a list of toothpastes.

message:
[BrandA, toothpaste, $12.98, 100g
 BrandB, toothpaste, $7.99, 80g
 BrandC, toothpaste, $1.99, 100g]

The list needs to be sent to specified destination topics based on the custom dynamic rules. The following items describe the rules:

Send the messages that start with BrandA to the BrandA-item-topic and BrandA-discount-topic topics.
Send the messages that start with BrandB to the BrandB-item-topic and BrandB-discount-topic topics.
Send other messages to the Unknown-brand-topic topic.

The following sample code shows the JSON format of the rules:

{
  "defaultTopic": "Unknown-brand-topic",
  "rules": [
    {
      "regex": "^BrandA",
      "targetTopics": [
        "BrandA-item-topic",
        "BrandA-discount-topic"
      ]
    },
    {
      "regex": "^BrandB",
      "targetTopics": [
        "BrandB-item-topic",
        "BrandB-discount-topic"
      ]
    }
  ]
}

dataclean_dynamicroute

Procedure

For the detailed procedure, see Content splitting. Set Function Template to Dynamic Routing dynamic_routing.

Content enrichment

Examples

In this example, IP address segments are enriched. The following code snippet shows the access logs of a service:

{
  "accountID": "164901546557****",
  "hostIP": "192.168.XX.XX"
}

The source of the IP address needs to be queried, and the mapping relationship needs to be stored in a MySQL database.

CREATE TABLE `tb_ip` (
    ->      `IP` VARCHAR(256) NOT NULL,
    ->     `Region` VARCHAR(256) NOT NULL,
    ->      `ISP` VARCHAR(256) NOT NULL,
    ->      PRIMARY KEY (`IP`)
    -> );

The following code snippet shows the processed message.

{
  "accountID": "164901546557****",
  "hostIP": "192.168.XX.XX",
  "region": "beijing"
}

dataclean_enrich

Procedure

For the detailed procedure, see Content splitting. Set Function Template to Content Enrichment transform_enrichment.

Content mapping

Examples

The following message contains the registration information about the employees of a company. The information includes confidential content, such as employee IDs and mobile phone numbers.

Jack, Employee ID 1, 131 1111 1111
Alice, Employee ID 2, 132 2222 2222
John, Employee ID 3, 133 3333 3333

The confidential content in the preceding message must be masked before the message is sent to the destination services. Example:

Ja**, Employee ID *, ***********
Alic*, Employee ID *, ***********
Joh*, Employee ID *, ***********

dataclean_projection

Procedure

For the detailed procedure, see Content splitting. Set Function Template to Content Mapping transform_projection.