This topic describes message queues in ApsaraMQ for RocketMQ. It covers their definition, model relationship, internal properties, version compatibility, and usage recommendations.
Definition
A queue is a container in ApsaraMQ for RocketMQ that stores and transmits messages. A queue is also the smallest storage unit for ApsaraMQ for RocketMQ messages.
All topics and lightweight topics in ApsaraMQ for RocketMQ consist of one or more queues. This architecture supports the horizontal splitting of queues and stream-based storage within each queue.
Queues serve the following main purposes:
Ordered storage
Queues are inherently ordered. Messages are written to storage in the order they enter the queue, which creates a natural order for messages within the same queue. The earliest message is at the head of the queue, and the latest message is at the tail. An offset is used to mark the position and order of each message in the queue.
Streaming operation semantics
The queue-based storage model of ApsaraMQ for RocketMQ lets you read any number of messages from any offset. This enables features such as aggregate reads and backtrack reads. Other messaging systems that do not use a queue-based storage model, such as RabbitMQ and ActiveMQ, do not provide these features.
Model relationship
The following figure shows the position and flow of a queue within the domain model of ApsaraMQ for RocketMQ:
By default, ApsaraMQ for RocketMQ provides a reliable message storage mechanism. All successfully sent messages are persistently stored in a queue. This mechanism, combined with calls from producer and consumer clients, achieves at-least-once delivery semantics.
The queue model of ApsaraMQ for RocketMQ is similar to the partition model of Kafka. In the ApsaraMQ for RocketMQ messaging model, a queue is a component of a topic. Although all message resources are managed at the topic and lightweight topic granularity, the actual operations are performed on queues. For example, a producer sends a message to a specific topic, but the message is delivered to a queue within that topic.
ApsaraMQ for RocketMQ implements horizontal scaling, both for scaling out and scaling in, by changing the number of queues.
Internal properties
Behavioral constraints
Each topic and lightweight topic uses one or more queues to store messages. The number of queues for each topic or lightweight topic depends on the message type and the region where the instance resides. The number of queues cannot be modified.
Version compatibility
The queue name property differs across versions of the ApsaraMQ for RocketMQ service:
Service versions 3.x and 4.x: A queue name is a trituple of {Topic Name}+{Broker ID}+{Queue ID} and is bound to a physical node.
Service version 5.x: A queue name is a globally unique string assigned by the cluster and is decoupled from physical nodes.
Therefore, during development, do not make assumptions about queue names or bind them to other operations. If you manually construct queue names in your code and bind them to other operations, you may encounter compatibility issues in which the queue name cannot be parsed after a service upgrade.
Usage recommendations
Set the number of queues based on business consumption
For ApsaraMQ for RocketMQ queues, follow the principle of using as few as necessary. Avoid arbitrarily increasing the number of queues.
An excessive number of queues in a topic can cause the following problems:
Cluster metadata bloat
ApsaraMQ for RocketMQ collects metrics and monitoring data at the queue granularity. An excessive number of queues can cause control plane metadata to bloat.
Excessive client load
In ApsaraMQ for RocketMQ, message reads and writes are performed on queues. An excessive number of queues can generate empty polling requests, which increases the system load.
Common scenarios for increasing queues
Increase queues for physical node load balancing
In ApsaraMQ for RocketMQ, the queues of a topic can be distributed across different service nodes. After you horizontally scale a cluster by adding nodes, you can add new queues on the new service nodes or migrate old queues to them. This ensures load balancing for cluster traffic.
Increase queues to scale ordered message performance
In ApsaraMQ for RocketMQ service version 4.x, message order is guaranteed only within a single queue. Therefore, the number of queues affects the concurrency of ordered messages. Increase the number of queues only when you encounter a system performance bottleneck.
Load balancing for unordered message consumption is independent of the number of queues
For instances of the ApsaraMQ for RocketMQ service 5.x series, consumer load balancing is upgraded to message-level granularity. Messages within the same queue can be consumed evenly by all consumers. Therefore, you do not need to worry about the number of queues affecting consumption concurrency. For more information about load balancing, see Consumer load balancing.