By RocketMQ Community
At the end of 2020, the much-anticipated Apache RocketMQ 4.8.0 was finally released. In this version, the community completed a large number of optimizations and bug fixes for RocketMQ. More importantly, this version improves the performance, stability, and functionality of the DLedger mode.
DLedger is a Raft-based CommitLog repository implementation in OpenMessaging. Starting from RocketMQ 4.5.0, RocketMQ introduces the DLedger mode to solve automatic failover in the Broker group. In version 4.8.0, the community comprehensively upgraded the RocketMQ DLedger mode.
RocketMQ 4.7.0 re-upgraded the synchronous dual-write architecture and improved the synchronous dual-write performance using the asynchronous pipeline mode. In RocketMQ 4.8.0, the community applied this improvement to the DLedger mode. The following figure shows how the broker processes message sending in DLedger mode. In the original architecture, the message processed by the SendMessageProcessor thread is only returned to the client after the Quorum has confirmed replication success. However, in the new version, CompletableFuture is used to asynchronously improve the process of message processing by the thread. The next request can be processed without waiting for confirmation from the Quorum. The ACK operation is confirmed by other threads, and the result is processed and returned to the client. The replication process is split and streamlined to reduce the long waiting time of threads and make full use of the CPU. By doing so, the throughput is greatly improved.
Batch is always an important way to optimize performance. In the new version, the "isEnableBatchPush = true" can be set to enable the batch replication of DLedger mode. The number of receiving and sending packets can be reduced by aggregating multiple pieces of data in the same packet and sending them together. This reduces system calls and context switching. Batch replication significantly improves the throughput under the pressure of data sending that may reach a bottleneck of packet receiving and sending. Significantly, batch replication in DLedger mode does not implement delayed accumulation and processing of a single packet. Therefore, there is no latency in sending a single message.
In addition to the preceding performance optimization, the community has also made some performance optimizations, such as locks and caches. This improves the performance of DLedger mode several times over.
The community used OpenMessaging-Chaos to perform a large number of Chaos tests on the RocketMQ DLedger mode to verify and test the reliability of DLedger mode that was supplementary to a wide variety of local tests on DLedger mode. OpenMessaging-Chaos is a test framework that uses fault injection to verify the consistency and high availability of various message platforms. In the OpenMessaging-Chaos test, clients concurrently send and receive messages to the clusters to be tested. Fault injection is performed at intervals on the clusters. Finally, the test results are given, including message loss, duplicate messages, and the average fault recovery time of the cluster. The performance of the DLedger mode in the following fault injection scenarios is verified using OpenMessaging-Chaos:
Let's take minor-kill fault injection as an example. RocketMQ broker in DLedger mode composed of five nodes is deployed for Chaos testing. Minor-kill fault injection randomly selects and kills a small number of nodes in the cluster. Due to the small number of nodes being killed, clusters can be restored within a short time even it is unavailable. This is convenient for testing the average restoration time of clusters.
During the test, four clients are set to send and receive messages to the parallel RocketMQ DLedger cluster. The time interval for fault injection is 100s. The cluster runs normally for 100s and then is injected with fault for 100s. The process is repeated and lasts 2,400s in total. After the test is completed, OpenMessaging-Chaos presents the test result and the latency graph. The following figure shows the latency of enqueue operations during the test.
In the figure above, the ordinate represents the time latency; the abscissa represents the test time; the green box represents the data sending success; the red box represents the data sending failure; the blue box represents the uncertainty of whether the data is added successfully; the gray part indicates the period of fault injection. Some fault injection periods cause the temporary unavailability of clusters, while some fault injection periods do not, and these results are reasonable. The network partition is random, so the cluster re-election can only be initiated when the leader node is included in a few killed nodes. However, even if the cluster re-election is triggered, the DLedger mode can still restore availability after a short time.
The following figure shows the given statistical results by OpenMessaging-Chaos after the test. It shows 110,000 pieces of data have been sent successfully with no data lost. Even if a fault occurs, the RocketMQ DLedger mode still complies with the At Least Once delivery semantics. In addition, RTOTestResult indicates that the clusters become unavailable three times during the 12 different failures, which is consistent with the latency graph above. Each cluster can be recovered within 30 seconds, and the average failure recovery time is about 22 seconds.
During the OpenMessaging-Chaos test, several hidden problems in the DLedger mode are found and fixed. This improved the fault tolerance capability of abnormal process crashes, slow nodes, symmetric and asymmetric network partitions, and network packet loss. These improvements further verify the reliability of the DLedger mode.
In the earlier version, it selected one from a group of brokers as the Leader at random. However, in the new version, preferredLeaderId can be configured to prioritize a node as the Leader. As shown in the following figure, preferred leader can be used to align DCs by deploying a broker group of DLedger mode in the three data centers. In this case, the service performance in DC1 is better. Thus, the Master is chosen preferentially to be deployed in DC1. In addition, the preferred leader can be used to implement the hybrid deployment of DLedger clusters to make full use of machine resources.
From RocketMQ 4.8.0, the DLedger mode supports sending messages in batches. Its support for the message types is consistent with the Master-Slave deployment.
In addition to the massive optimizations to the DLedger mode, the RocketMQ version made 25 improvements, 11 bug fixes, and 16 document and code format optimizations. According to partial statistics, nearly 40 RocketMQ community members contribute. By this opportunity, I appreciate everyone's active participation. Users, manufacturers, and developers are welcomed to participate in the construction of RocketMQ. It is never too late to join the Apache RocketMQ community!
The Development Trends of Six Major Container Technologies in 2021
506 posts | 48 followers
FollowAlibaba Cloud Native - October 12, 2024
Alibaba Cloud Native - June 7, 2024
Alibaba Cloud Native Community - February 1, 2023
Alibaba Cloud Native Community - May 15, 2023
Alibaba Cloud Native Community - October 26, 2023
Alibaba Cloud Native Community - March 20, 2023
506 posts | 48 followers
Follow
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
AliwareMQ for IoT
A message service designed for IoT and mobile Internet (MI).
Learn More
ChatAPP
Reach global users more accurately and efficiently via IM Channel
Learn More
Message Queue for RabbitMQ
A distributed, fully managed, and professional messaging service that features high throughput, low latency, and high scalability.
Learn MoreMore Posts by Alibaba Cloud Native Community
