This topic describes the features and limits of the Message Integration module of the ApsaraMQ for RocketMQ console. This topic also introduces the basic terms that are used in Message Integration.
Background information
In the process of digital transformation, enterprises face various pain points, such as the lack of a centralized method to integrate information, inefficient data transfer and integration caused by various data formats and types, the lack of a service to share data and backend services, and the lack of a secure channel to connect cloud-based and on-premises resources. To resolve the pain points, Alibaba Cloud provides a full-stack message and data integration module called Message Integration. The module helps you simplify the message integration process and supports cloud-based and on-premises integration across regions.
Message Integration provides low-code and full-stack event stream processing for messaging services. Focusing on scenarios such as message integration, data connection, data processing, and service integration, Message Integration provides a visualized user interface that allows you to easily create and orchestrate integration tasks. Message Integration also allows you to connect devices across regions, instances, and applications. This helps reduce costs for message integration and development.
Features
Various data sources
Message Integration aims to connect data that is stored in and across clouds. Message Integration allows you to synchronize data from the logs and self-managed applications of various services in the cloud to a destination service. Data synchronization across clouds is also supported. You can use Message Integration to synchronize data from logs, databases, and messaging middleware.
Data cleansing and data outflow
Message Integration provides powerful, UI-based extract-transform-load (ETL) capability. You can use the capability to cleanse and format data that is received by message queues and convert the format of the data. Message Integration also allows you to dump data to downstream applications after ETL processing.
Serverless custom processing
By leveraging the custom coding capability provided by Function Compute, a serverless computing service provided by Alibaba Cloud, Message Integration allows you to write custom business logic and complete custom data processing and distribution.
Limits
The following table describes the limits of Message Integration.
Item | Description |
Number of tasks | The total number of message inflow and outflow tasks in a region cannot exceed 20. If you want to increase the number of tasks, submit a ticket. |
Task name | The name can contain letters, digits, and hyphens (-), and must start with a letter or a digit. The name can be up to 127 characters in length. If the name that you specify is more than 127 characters in length, the system truncates the value. |
Event pattern | In the stringExpression pattern, each field can contain up to five expressions in the map data structure. |
Event transformation |
|
Data loss in extreme cases | For more information, see Retry policies and dead-letter queues. |
Message Inflow (Source)
Message Inflow, or Source, is a component of Message Integration. Message Inflow allows you to integrate data, including messages, logs, relational data, and non-relational data, between multiple data sources in a flexible, rapid, and non-intrusive manner. Message Inflow can help implement data integration solutions across data centers and clouds, and provides features to help you operate, manage, and monitor the integrated data. The major capabilities of Message Inflow are to synchronize data of various types to messaging services, and manage and filter data from different sources.
Message Outflow (Sink)
The main capability of Message Outflow, or Sink, is to export data from messaging services to various destination services. Message Integration reliably distributes messages from messaging services and cleanses and filters the message data during distribution.
Data Processing (Transform)
Data Processing, or Transform, is an important component of Message Integration. This capability is optional when you create a message inflow or outflow task. Message Integration provides seven matching modes and five transformers based on the data processing capability provided by EventBridge and the custom definition capability provided by Function Compute to efficiently transform, process, and analyze messages. The matching modes are fixed-value matching, prefix matching, suffix matching, exclusion-based matching, numeric-value matching, array matching, and combined-condition matching. The transformers are complete event, partial event, constant, template, and Function Compute template.
Task
A task is a resource entity that runs Message Integration. A task is also the method that is used to implement Message Integration. In most cases, a task consists of a source, a destination, a filtering rule, and a transformation rule. The underlying resource of a task is the event stream feature of EventBridge.
When you create a task, you must specify the resources and the method that is used to process data. The configuration of resources is an important part of a task and includes the configuration of the source and destination. For information about Data Processing, see Data Processing (Transform).
After you create a task, you cannot change the resource type that you configured for the task.