You can use the Global Replicator feature of ApsaraMQ for RocketMQ to synchronize messages and consumer progress across regions and instances. This topic describes the Global Replicator feature.
Introduction
Global Replicator allows you to synchronize messages and consumer progress across regions and instances. You can use it in scenarios such as data synchronization between Apache RocketMQ and ApsaraMQ for RocketMQ clusters, geo-disaster recovery, and active geo-redundancy.
Cross-region data synchronization
Global Replicator supports message and consumer progress synchronization between clusters in different regions to meet requirements on low latency and high availability in global business scenarios.
Seamless cross- instance integration
Compatible with open source Apache RocketMQ and ApsaraMQ for RocketMQ clusters, Global Replicator can flexibly synchronize messages and consumer progress to implement data flow in a hybrid cloud or multi-cloud architecture.
Support of multiple message types
Global Replicator supports synchronization of the following types of messages: normal messages, ordered messages, scheduled or delayed messages, and transactional messages.
Support of message and consumer progress synchronization
If an error occurs in the source instance, you can use the quick synchronization feature to synchronize the consumer progress of the destination group to the source instance. This reduces the number of duplicated messages consumed.
Support of one- and two-way synchronization tasks
By default, two-way data synchronization tasks can prevent data from being cyclically synchronized.
Global Replicator consumes the read and write resources of ApsaraMQ for RocketMQ. Before using the feature, evaluate the computing specifications.
Figure 1. Message synchronization of ApsaraMQ for RocketMQ
Scenarios
All clusters involved in Global Replicator are independent instances that provide read and write services. Messages are synchronized between instances in an asynchronous manner. Global Replicator can be used in the following scenarios.
Data synchronization (one-way)
Business deployed in each independent unit (business environment) is processed on the server closest to the unit without cross-network data access. This prevents timeout issues caused by networks. You can use the one-way synchronization capability of Global Replicator to synchronize data in different regions and instances (open source Apache RocketMQ and ApsaraMQ for RocketMQ clusters) to the central region for unified processing.
Industries: banking, securities, insurance
Architecture: multiple units, single center
Figure 2. Architecture for data aggregation in ApsaraMQ for RocketMQ
Geo-disaster recovery (one-way)
The one-way synchronization capability of Global Replicator can implement cross-region and cross-instance high availability. When a fault occurs in a data center or a region, you can use the one-way synchronization capability to quickly restore business.
Industries: all
Architecture: two regions, two data centers
Figure 3. Architecture for geo-disaster recovery in ApsaraMQ for RocketMQ
In normal cases, you do not need to run applications in the secondary region or center. This helps reduce resource consumption and save costs. If an exception occurs, you can start the applications in the secondary region or center, then use the consumer offset resetting feature of ApsaraMQ for RocketMQ instances to consume a minimum number of historical messages on the source instance. We recommend that you implement message idempotence in your business logic to reduce impacts caused by message duplication.
Active geo-redundancy (two-way)
If you want to deploy your business in multiple units and ensure business continuity at the same time, you can use the two-way synchronization capability of Global Replicator to implement cross-region and cross-instance high availability.
Industries: finance, energy, and other industries related to the livelihood of people
Architecture: two regions, two data centers
Figure 4. Architecture for active geo-redundancy in ApsaraMQ for RocketMQ
When you use the Global Replicator feature to synchronize data, tags are added to messages. No extra development is required for producer applications, and consumer applications can consume messages containing specific tags by using the SQL-92 filtering method provided by ApsaraMQ for RocketMQ.
If an exception occurs, the disaster-recovery capabilities of the business can ensure that the overall business runs as expected. In this case, producer applications can receive all messages and deliver the messages to the ApsaraMQ for RocketMQ instance on which no exceptions occur. However, you must modify the filtering conditions on the consumer applications. You can use the consumer offset resetting feature of the ApsaraMQ for RocketMQ instance to consume a minimum number of messages from other regions. We recommend that you implement message idempotence in the business logic to reduce impacts caused by message duplication.
Technical architecture
Global Replicator synchronizes messages in an asynchronous manner to implement cross-region data synchronization via connectors. Connectors feature the scaling of distributed systems in seconds, rule-based tagging, and resumable upload and download. Networks between instances that reside in different regions are connected by using Cloud Enterprise Network (CEN). Messages in instances are synchronized with a latency of seconds.
Figure 5. Architecture for cross-region synchronization of data in ApsaraMQ for RocketMQ
Message request description
ApsaraMQ for RocketMQ instances involved in Global Replicator are independent of each other and have read and write capabilities. The instance that an application accesses is determined by the instance endpoint that you configured for the application. When messages are synchronized between two instances, tags are added to the messages based on the tag key and value specified in the user attributes. You can use the tags to filter messages on the consumer application to determine whether to receive either full data or only local data. The following figure shows how to add tags to messages. For more information, see Usage notes.
Figure 6. Adding tags to messages during the synchronization of messages in ApsaraMQ for RocketMQ
Benefits
Low-code development: Messages are synchronized between instances by using Global Replicator. This reduces the development workload.
Flexible configuration: Global Replicator allows you to configure one- or two-way message synchronization between instances and add tags to messages for flexible message selection.
Low-latency synchronization across regions: EventBridge is used to implement message synchronization. This ensures that messages can be stably and flexibly synchronized with a latency of only seconds.
Billing
Global Replicator depends on EventBridge for synchronization. For information about the billing of EventBridge, see Billing.
Global Replicator consumes the read resources of the source ApsaraMQ for RocketMQ instance and the write resources of the destination ApsaraMQ for RocketMQ instance. For information about the billing of ApsaraMQ for RocketMQ, see Billing.
Supported regions, editions, and versions
Supported regions: Global Replicator is available in all supported regions of ApsaraMQ for RocketMQ. For more information, see Region mappings between the source and destination clusters.
Supported editions and versions:
ApsaraMQ for RocketMQ 4.0 instances.
NoteThe Standard Edition only supports one-way synchronization.
ApsaraMQ for RocketMQ 5.0 instances.
You can use Global Replicator to synchronize data between ApsaraMQ for RocketMQ Enterprise Platinum Edition instances and Apache RocketMQ clusters. The version of Apache RocketMQ must be 4.4.0 or later.
You can use Global Replicator to synchronize data from one ApsaraMQ for RocketMQ instance to multiple ApsaraMQ for RocketMQ instances.