RocketMQ 5.0 POP Consumption Model

By Kaiyi & Yuntian
Proofreading: Baiyu

Major features are becoming available with the release of the RocketMQ 5.0 preview. POP Consumer, a key feature of 5.0, presents a new consumer model. It is lightweight, stateless, and no queue-exclusive. It is friendly to message backlog and Streaming consumption scenarios. Before introducing POP Consumer, let's review the most commonly used Push consumers.

Push Consumer

People familiar with RocketMQ will be familiar with Push Consumer. Client-side consumption generally uses this consumption mode, and it is relatively simple to use. All we need to do is set up and write the business logic in the callback method ConsumeMessage and start the client application to consume messages.

public class PushConsumer {

    public static void main(String[] args) throws InterruptedException, MQClientException {
        DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_JODIE_1");
        consumer.subscribe("test_topic", "*");
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
        consumer.registerMessageListener(new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                System.out.printf("%s Receive New Messages: %s %n", Thread.currentThread().getName(), msgs);
                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
            }
        });
        consumer.start();
        System.out.printf("Consumer Started.%n");
    }
}

How does Push Consumer consume messages?

The Consumer only receives the message if the Producer sends it to the Topic first. The Producer uses polling to send messages to each Queue separately. Generally, there is more than one consumer. When the client starts, it will be SLB in the Topic and Consumer group dimension and assign a Queue to be processed to each client. During the SLB process, each client can get all ConsumerID and queues and sort them. Each client uses the same Load Balancing Algorithm, such as the uniform allocating algorithm, so each client calculates which Queues it needs to consume. SLB will be triggered whenever the consumer increases or decreases. Therefore, we can implement dynamic expansion through the RocketMQ SLB mechanism to improve the client's ability to send and receive messages.

Can the number of clients be increased all the time to improve the consumption capacity? Of course not, because the number of Queues is limited. Once the number of clients reaches the number of Queues, expanding new nodes fails to improve the consumption capacity because some nodes cannot be allocated to Queues.

After client-side load balancing assigns Queue to the client, the client will keep pulling messages to the broker and consume at the client. Isn't it a Push client? How can it be that the client pulls messages to the broker? Shouldn't it be that the broker pushes messages to the client? This is an interesting point. In RocketMQ, all messages are consumed by client pulling whether it is Push Consumer, Pull Consumer, or POP Consumer (to be introduced later). Push Consumer only makes us feel pushed by Broker through encapsulation at the client API level.

After the client-side load balancing and pulling messages, the client can consume messages normally.

The complete processing logic of Push Consumer can be seen in the figure above.

First, the client Rebalance determines which consumer clients process which Queues and then pulls messages through the PullMessageService service. After pulling the messages, ConsumeMessageConcurrentlyService submits a consumption request to the message consumption thread pool. Then, call the callback method ConsumeMessage, where the message processing service can be obtained. Finally, the consumption updates the local offset and reports the offset to the Broker. If the consumption fails (exception throwing, timeout, etc.), the client will send sendBack to tell the broker which messages failed to be consumed. The broker sends the failed messages to the delayed queue and then puts them into the retry topic after the delay. The client consumes the retry topic to complete the retry. The advantage is that the consumption of normal messages will not be affected by failed messages. If you want to know the details, you can go to GitHub to download the source code and compare this figure to see the code processing flow.

The introduction of Push Consumer gives us a certain understanding of the principle of Push Consumer. RocketMQ clients do many things, such as SLB, message pulling, consumer offset management, and sendBack after failed consumption. This is unfriendly to multilingual support. People that have participated in multilingual development will feel the same way. It is difficult to transplant so much logic to different languages. This increases the difficulty of upgrading and O&M on the client side.

Do we think about whether we can slim down the client and move part of the logic from the client to the Broker? Of course, it is possible. From the previous introduction, we know when the Push Consumer client is responsible for balancing, the information SLB needs and all ConsumerId were originally obtained by the client from the Broker. All Queue information can be obtained by Broker through nameServer. There is no big difference between the client and the broker side to call the algorithm, so porting Rebalance to the Broker is a good choice. The Broker SLB can achieve the same effect as the client SLB. The client logic will be reduced, the multi-language implementation will be simpler, and the subsequent upgrade operation and maintenance will be more controllable. In addition, since the Broker has global information compared with the client. It can do some more interesting things, such as when responsible for balancing, executing SLB according to the backlog of Queue and assigning Queue from stressful clients to other clients for processing.

POP Consumer

We learned some features of Push Consumer through the introduction of Push Consumer.

Queue Exclusive: Each queue on Broker can be allocated to one Push Consumer machine with the same consumer group.
Update Offset after Consumption: Pull a batch of messages to the local queue cache for each pull request. The offset is committed only when the local consumption is successful.

The preceding features may cause some problems. For example, an abnormal machine hangs on the client, which leads to the accumulation of allocation queue messages, thus failing to consume.

Push consumer consumption in RocketMQ is unfriendly when the machine is abnormal. If the client machine hangs but connects to the Broker, the client Rebalance will allocate a consumption queue to the hanging machine. When the consumption speed of the hanging machine is slow or cannot consume, there will be consumption accumulation. There are also unavoidable problems, such as consumption delay caused by Rebalance of the client when the server Broker is released. It is shown in the following figure:

When the Push Consumer 2 machine hangs, Q2 on the Broker is piled up. We deal with this problem by finding this machine to restart it or make it offline. We need to ensure that the business is not affected by abnormal machines. However, if the queue is squeezed to some extent, the machine may be unable to quickly catch up with the consumption progress after recovering, which is also limited by the ability of Push Consumer.

Let's summarize some pain points of Push Consumer:

The Client-side is bloated with heavy logic and is multilingual-unfriendly.
Rebalance, such as upgrading, releasing, and restarting from the client or Broker side, may cause a consumption squeeze.
Queue Occupancy: A single queue is bound to a single consumer, and the consumption capacity of a single queue fails to be scaled out.
The machine is prone to hang, resulting in squeezing.

Based on the preceding issues, RocketMQ 5.0 implements a new consumption model-POP Consumer.

POP Consumer can solve the preceding stability problem and relieve the extended capability of queue occupancy.

Let's take a brief look at how POP Consumer consumes messages:

The POP Client issues a POP request message from the Broker's queue, and the Broker returns messages. In the massage system attribute, there is an important one called POP_CK, which is the handler of a message. A message can be located through one handler. After the message consumption is successful, the POP client sends ackMessages and passes the handler to the broker to confirm the message consumption is successful.

For message retry, when POP sends out a message, this message will enter an invisible time, during which time it will not be POP again. If the message consumption is not confirmed through ackMessage during this invisible time, it will be visible again after the invisible time has passed.

In addition, the policy is a gradient delay time, and the retry gap gradually increases for the retry of the message. There is another changeInvisibleTime that can modify the invisible time of the message.

As shown in the figure above, the message would have been visible again in the middle of the time, but we used the changeInvisibleTime in advance to extend the invisible time and delay the visible time of this message. When the user service code returns reconsumeLater or throws exceptions, we can modify the next visible time based on the number of retries through changeInvisibleTime. In addition, if the consumption RT exceeds 30 seconds (the default value which can be modified), the broker will also put the message into the retry queue.

Moreover, POP consumption is stored and controlled by brokers. POP consumption can be consumed by multiple clients in the same queue, as shown in the following figure:

The three clients do not need Rebalance to allocate Queues. Instead, they all use POP to request all brokers to obtain messages for consumption. Even if POP Consumer 2 hangs, its internal messages will be consumed by POP Consumer 1 and POP Consumer 3. Then, the problem of consumption accumulation caused by hanging machines is solved.

As seen in the overall process, POP consumption can avoid the consumption delay caused by Rebalance, and the client can consume all the queues of the broker. This way, the problem of accumulation caused by machine hanging can be avoided.

At the same time, the expansion capability is improved. POP Consumer can consume all queues under the same topic. Compared with Push Consumer, the limit that each Queue must be rebalanced to one client consumption is lifted. The number of Push Consumer clients must be less than or equal to the number of Queues. POP consumers can break this limit. Multiple POP consumers can consume the same queue.

Broker Implementation

How is POP Consumer implemented on the broker side?

After a POP Consumer pulls a message, it locks it in the Queue dimension to ensure that only one client can pull messages to the same Queue at the same time. After the message is obtained, the checkPoint information will be stored in the broker. The checkPoint information mainly includes messages, such as Topic, ConsumerGroup, QueueId, offset, POPTime, msgCout, and reviveQueueId. The checkPoint information is preferentially saved to the buffer, waiting for the ack message and receiving the ack message replied by the client within a period. The corresponding checkPoint information is removed from the buffer, and the consumption progress is updated to identify that the message consumption is successful.

When the checkPoint message waits in the buffer and does not get the ack message, it will be cleared out of the buffer and will send a ck msg to the store. The ck msg will be sent to the delay queue SCHEDULE_Topic_XXXX first and will enter the REVIVE_LOG Topic after the delay is completed. The REVIVE_LOG Topic is the topic of ck msg and ack msg stored to be processed. POPReceiveService pulls the REVIVE_LOG Topic message into a map. If ck has a corresponding ack, the consumer offset of REVIVE_LOG will be updated to identify the ck msg whose message consumption has been completed but not been confirmed after a timeout. The real message corresponding to the ck msg will be queried, and this message will be put into the retry topic, waiting for the client to consume. POP Consumer will probably consume the message in the retry topic when it is normally consumed. You can see from this design that the common design of RocketMQ is used to implement business logic, transactional messages, and scheduled messages through some internal topics.

Let's briefly summarize the benefits of POP Consumer:

Stateless: The offset information is maintained by the Broker, and the client is not bound to the Queue.
Lightweight: The client only needs to receive and confirm messages.
No Queue-Exclusive: A queue is no longer bound to the client.
Multilingual: It is convenient for multilingual transplantation.
Controllable Upgrades: The logic converges to the Broker, so the upgrade is convenient and controllable.

POP & Push Integration

Since POP has so many advantages, can we use POP to solve some problems of Push? As we mentioned earlier, when a queue has accumulated a lot of consumer problems, Push Consumer is limited by the consumption-ability of a single consumer and cannot quickly catch up with the consumption progress. The delay will be high. The core problem is the limitation of single-queue and single-consumer, which makes the consumption capacity unable to scale out.

We hope that when a queue is piled up, we can switch to POP mode through the POPAPI form and have the opportunity for multiple consumers to consume the queue together and catch up with the progress. We also achieved this in the 5.0 version.

POP/Push Mode Switching Method

You can switch in two ways:

1. Command Line

mqadmin setConsumeMode -c cluster -t topic -g group -m POP -n 8

2. Code Switching

    public static final String CONSUMER_GROUP = "CID_JODIE_1";
    public static final String TOPIC = "TopicTest";

    // Or use AdminTools directly: mqadmin setConsumeMode -c cluster -t topic -g group -m POP -n 8
    private static void switchPop() throws Exception {
        DefaultMQAdminExt mqAdminExt = new DefaultMQAdminExt();
        mqAdminExt.start();

        ClusterInfo clusterInfo = mqAdminExt.examineBrokerClusterInfo();
        Set<String> brokerAddrs = clusterInfo.getBrokerAddrTable().values().stream().map(BrokerData::selectBrokerAddr).collect(Collectors.toSet());

        for (String brokerAddr : brokerAddrs) {
            mqAdminExt.setMessageRequestMode(brokerAddr, TOPIC, CONSUMER_GROUP, MessageRequestMode.POP, 8, 3_000);
        }
    }

We can see that POP Consumer and Push API are unified and simple to use through the following POP Consumer Demo. Compared with Push API, it is one more step of consumption mode switching.

Push & POP Retry Queue Differences

When using POP consumption mode, we only need to switch modes based on Push API. We still need to do some processing for the Broker. Retry queue is the main thing that needs to be handled.

Push and POP modes process retry queues differently:

Retry processing of Push
The server has a %RETRY%ConsumerGroup queue.
The client will have a pull task to pull messages from this queue.
Retry processing of POP
The server has a retry queue named %RETRY%ConsumerGroup_Topic for each topic.
The client does not have a pull task specifically for the retry queue, and every ordinary POP request has a certain probability of consuming the corresponding retry queue.

After the mode is switched, the messages in the retry of the old mode need to be processed. Otherwise, the messages will be lost.

Push & POP Switching

From Push to POP

Switch the normal queue to POP mode
The POP request of the normal queue will process the corresponding POP retry queue.
For the Push retry queue, we keep the pulling task of the original push retry queue and work in Push mode.

From POP to Push

Switch the normal queue to Push mode
Push retry queues have corresponding pull tasks
In the previous POP retry queue, we automatically created a pull task on the client and pulled it in Push mode. Note: The pulling task here only pulls the retry queue of POP.

In summary, we will handle the retry queue without mode switching.

Summary

POP is a new consumption mode that solves some pain points of Push mode, makes the client stateless and lighter, and converges the consumption logic to the Broker. Meanwhile, it is friendly to multilingual support. It has also been integrated with Push at the API level, inheriting the simplicity and ease of use of Push API and realizing free switching between Push and POP.

Community

RocketMQ 5.0 POP Consumption Model

Push Consumer

POP Consumer

Broker Implementation

POP & Push Integration

POP/Push Mode Switching Method

Push & POP Retry Queue Differences

Push & POP Switching

From Push to POP

From POP to Push

Summary

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

ApsaraMQ for RocketMQ

AliwareMQ for IoT

Message Queue for Apache Kafka

Message Queue for RabbitMQ