ApsaraMQ for RocketMQ supports clustering consumption and broadcasting consumption. You can set the consumption mode of consumers in a cluster to clustering or broadcasting on clients to specify whether a message is consumed by only one consumer or all consumers in the cluster.
Background information
ApsaraMQ for RocketMQ is a messaging system that is based on the Publish-Subscribe pattern. In ApsaraMQ for RocketMQ, a consumer subscribes to topics to obtain and consume messages.
Multiple consumers that are identified by the same group ID belong to the same consumer cluster and must use the same consumption logic and tags. For more information, see Subscription consistency.
Clustering consumption
Basic concept
In clustering consumption mode, multiple consumers in a group consume messages from a specific topic. ApsaraMQ for RocketMQ delivers messages to different consumers and each message is consumed by only one consumer.
Scenarios
Message load balancing: If a large number of messages are processed in your business, you can use the clustering consumption mode to distribute messages to multiple consumers to implement load balancing. This way, each consumer processes only a small number of messages. This helps improve message processing efficiency and throughput.
Distributed data processing: For scenarios in which a large amount of data needs to be processed, you can use the clustering consumption mode to distribute data to multiple nodes. This way, each node processes only a small amount of data. This improves the processing rate.
Usage notes
In clustering consumption mode, a failed message is not necessarily redelivered to the same machine.
Make sure that you specify the same consumption mode for consumers in the same group. If you specify different consumption modes for consumers in the same group, the broadcasting consumption mode takes effect.
For example, you create a group that consists of five consumers. If you specify the clustering consumption mode for two consumers in the group and the broadcasting consumption mode for the other three consumers in the group, the broadcasting consumption mode is used for all five consumers.
Broadcasting consumption
Basic concept
In broadcasting consumption mode, when multiple consumers in a group consume messages in a specific topic, ApsaraMQ for RocketMQ delivers each message to all consumers in the cluster. This ensures that each message is consumed by each consumer at least once.
This mode is suitable for scenarios in which multiple consumers concurrently process the same group of messages, such as order processing.
Scenarios
Notifications and announcements: When an important announcement or notification is released, the system can use the broadcasting consumption mode to send the notification or announcement to all users who subscribe to the message. This ensures that each user receives the notification or announcement in a timely manner.
Real-time data synchronization: In some cases, you want to synchronize specific data in a distributed system in real time to maintain data consistency among multiple nodes. You can use the broadcasting consumption mode to send the notification about the data change to all nodes to which you want to synchronize the data change. This way, data on each node can be consistent.
Usage notes
In broadcasting consumption mode, ApsaraMQ for RocketMQ ensures that a message is consumed by consumers at least once, but does not redeliver messages that fail to be consumed. You need to manually handle consumption failures.
In broadcasting consumption mode, a consumer starts message consumption from the latest message each time the consumer is restarted. A consumer automatically skips the messages that are sent to the ApsaraMQ for RocketMQ broker when the consumer is stopped. Proceed with caution when you use this mode.
In broadcasting consumption mode, each message is repeatedly processed by a large number of consumers. We recommend that you use the clustering consumption mode in most cases.
Make sure that you specify the same consumption mode for consumers in the same group. If you specify different consumption modes for consumers in the same group, the broadcasting consumption mode takes effect.
For example, you create a group that consists of five consumers. If you specify the clustering consumption mode for two consumers in the group and the broadcasting consumption mode for the other three consumers in the group, the broadcasting consumption mode is used for all five consumers.
Differences between clustering consumption and broadcasting consumption
The availability of features varies based on the consumption mode. The following table lists the features that are available in clustering consumption mode and the features that are available in broadcasting consumption mode.
Feature | Clustering consumption mode | Broadcasting consumption mode |
TCP client SDKs |
|
|
HTTP client SDKs |
|
|
Ordered messages |
|
|
Consumer offset resetting |
|
|
Message retry |
|
|
Query and alerting of accumulated messages |
|
|
Subscription query |
|
|
Consumer progress | Maintained on ApsaraMQ for RocketMQ brokers.
| Maintained on clients. The probability that a message is repeatedly consumed in broadcasting consumption mode is higher than that in clustering consumption mode. Each time a consumer is restarted, the consumer starts consumption from the latest message. |
Configure the clustering consumption mode and the broadcasting consumption mode
ApsaraMQ for RocketMQ allows you to change consumption modes on clients. To change the consumption mode, you must configure the corresponding parameter in the code of the SDK that subscribes to messages. If you do not configure the parameter, the clustering consumption mode is used.
TCP client SDKs
Sample code for the SDK for Java
Clustering consumption
// Set the MessageModel parameter to CLUSTERING. If you do not configure this parameter, the clustering consumption mode is automatically used. properties.put(PropertyKeyConst.MessageModel, PropertyValueConst.CLUSTERING);Broadcasting consumption
// Set the MessageModel parameter to BROADCASTING. properties.put(PropertyKeyConst.MessageModel, PropertyValueConst.BROADCASTING);
Sample code for the SDK for C or C++
Clustering consumption
// Set the MessageModel parameter to CLUSTERING. If you do not configure this parameter, the clustering consumption mode is automatically used. factoryInfo.setFactoryProperty(ONSFactoryProperty:: MessageModel, ONSFactoryProperty::CLUSTERING);Broadcasting consumption
// Set the MessageModel parameter to BROADCASTING. factoryInfo.setFactoryProperty(ONSFactoryProperty:: MessageModel, ONSFactoryProperty::BROADCASTING);
Sample code for the SDK for .NET
Clustering consumption
// Set the MessageModel parameter to CLUSTERING. If you do not configure this parameter, the cluster consumption mode is automatically used. factoryInfo.setFactoryProperty(ONSFactoryProperty.MessageModel, ONSFactoryProperty.CLUSTERING);Broadcasting consumption
// Set the MessageModel parameter to BROADCASTING. factoryInfo.setFactoryProperty(ONSFactoryProperty.MessageModel, ONSFactoryProperty.BROADCASTING);
HTTP client SDKs
By default, HTTP client SDKs support the clustering consumption mode. You do not need to configure a parameter to specify the consumption mode.
FAQ
Why does the clustering consumption mode not take effect after I specify it?
The issue may occur due to one of the following reasons:
The messages that you process are ordered messages. Ordered messages support only clustering consumption.
The SDK that you use is of the HTTP client type. HTTP clients support only clustering consumption.
Can I specify different consumption modes for different consumers in a group?
No, you cannot specify different consumption modes for different consumers in a group.
Make sure that you specify the same consumption mode for consumers in the same group. If you specify different consumption modes for consumers in the same group, the broadcasting consumption mode takes effect.
References
For information about the complete sample code for messaging, see Overview.