By Qingshan Lin (Longji)
The development of message middleware has spanned over 30 years, from the emergence of the first generation of open-source message queues to the explosive growth of PC Internet, mobile Internet, and now IoT, cloud computing, and cloud-native technologies.
As digital transformation deepens, customers often encounter cross-scenario applications when using message technology, such as processing IoT messages and microservice messages simultaneously, and performing application integration, data integration, and real-time analysis. This requires enterprises to maintain multiple message systems, resulting in higher resource costs and learning costs.
In 2022, RocketMQ 5.0 was officially released, featuring a more cloud-native architecture that covers more business scenarios compared to RocketMQ 4.0. To master the latest version of RocketMQ, you need a more systematic and in-depth understanding.
Today, Qingshan Lin, who is in charge of Alibaba Cloud's messaging product line and an Apache RocketMQ PMC Member, will provide an in-depth analysis of RocketMQ 5.0's core principles and share best practices in different scenarios.
This course will review the background and development of RocketMQ, and how it has evolved to version 5.0 in response to cloud-native, IoT, and real-time data trends. You will learn about RocketMQ 5.0's cloud-native architecture and integrated architecture from a technical perspective. Then, we will introduce RocketMQ 5.0's capabilities and key technical principles in different business scenarios, including business messages, stream processing, IoT, and event-driven scenarios for the cloud era. I hope you will gain a new understanding of RocketMQ after taking this course. Let's start learning about Apache RocketMQ today.
In this course, you will learn about the background of RocketMQ and why Alibaba developed RocketMQ despite the existence of many popular message queues at the time. You will also learn about the driving forces behind RocketMQ 5.0's evolution and why it adopted certain technologies to expand its business capabilities.
This course is divided into three parts. First, I will help you review the evolution of message queues and successful ones that emerged in different times. Then, we will revisit the birth of RocketMQ and introduce its background and advantages. Finally, we will analyze the opportunities and challenges in the cloud era and help you understand the evolution of RocketMQ 5.0.
The operating system, database, and middleware are the three pillars of basic software, and message queues are one of the most classic middleware, with a history of over 30 years. Its development has gone through the following stages:
The first stage , before 2000, saw the birth of The Information Bus, the first message queue, which introduced the publishing and subscription mode to solve software communication problems. In the 1990s, international commercial software giants like IBM, Oracle, and Microsoft launched their own message queues, with IBM MQ being the most representative. This type of commercial message queue typically used high-end hardware and software, with a standalone architecture.
The second stage, from 2000 to 2007, witnessed the emergence of the first generation of open-source message queues, which led to the birth of JMS and AMQP standards and their implementations, ActiveMQ and RabbitMQ. Open-source greatly promoted the popularity of message queues, lowered the threshold of use, and made technology universal, gradually becoming the standard for enterprise-level architecture. Compared to today, these message queues were mainly used for traditional enterprise-level applications and small-traffic scenarios, with relatively weak horizontal scaling capabilities..
The third stage, from 2007 to 2017, was marked by the explosive development of PC Internet and mobile Internet. Traditional message queues couldn't withstand the massive traffic and data transmission services of hundreds of millions of users, leading to the birth of Internet message middleware. Its core capability is to fully adopt a distributed architecture and have strong horizontal scaling capabilities. This type of middleware includes open-source Kafka and RocketMQ, as well as closed-source Taobao Notify. The birth of Kafka also extended message middleware from the message domain to the stream domain and from asynchronous decoupling scenarios to stream storage and stream computing scenarios.
The fourth stage, from 2014 to today, has been shaped by the new wave of cloud computing, IoT, and big data.
Next, let's return to RocketMQ and Alibaba's core business scenario: e-commerce, to see how RocketMQ was born and what core values it creates for the business.
Alibaba's e-commerce system was originally a huge single-boulder application, which could not meet the rapid development of Taobao and Tmall in terms of research and development efficiency and stability. To solve this problem, Taobao and Tmall launched a largest-scale architecture upgrade in 2008, splitting single applications into distributed applications and abstracting the common base of Taobao and Tmall: business mid-end, including trading center, commodity center, buyer center, etc. Under the business mid-end, Alibaba middleware was also born (the initial three pillars include: message, RPC, and distributed data layer), and RocketMQ is one of them.
At that time, there were already many commercial or open-source message queues in the industry, such as IBM MQ, ActiveMQ, and RabbitMQ. However, without exception, these message queues were born in traditional enterprise-level application scenarios and could not withstand the Internet's demanding requirements for high concurrency and unlimited expansion. Take RabbitMQ as an example. The queue traffic and storage load of RabbitMQ are standalone, which cannot meet the needs of business scale-out. At that time, another message queue with unlimited scale-out capability was Kafka, but it was mainly used in log scenarios and lacked stability. In addition, it preferred a simple log message queue and could not meet the demands of e-commerce for complex message features, such as message filtering and delayed messages.
Another problem is that, the traditional message queue cannot meet the e-commerce business's requirements for distributed consistency. After application asynchronous decoupling is implemented through the message queue, the e-commerce business also needs to ensure that different upstream and downstream applications reach a final agreement on the order status, otherwise, a large number of dirty data will be generated and business errors will be caused. A large-scale e-commerce system requires both high performance and consistency. Traditional distributed transaction technologies are helpless. For example, IBM MQ can use XA transactions to meet the functional demands of distributed consistency. However, the delay and cost caused by XA are unbearable for massive Internet traffic.
To meet the e-commerce businesses' requirements for high performance, consistency, and unlimited expansion of the message queue, a self-research message queue became the only way out at that time. In this context, the Internet message queue RocketMQ came into being.
To support ultra-large-scale and complex e-commerce businesses, RocketMQ focuses on four aspects of construction and forms four major advantages.
1. Support for ultra-large-scale and complex businesses with rich message features
Every large Internet company has a main business and expands its business capabilities with the main business as the center. Alibaba e-commerce is an e-commerce operating system built around transaction events. Each transaction event triggers different businesses, and different business segments focus on different types of transaction events. The SQL subscription of RocketMQ provides flexible message filtering capabilities to meet the demands of downstream consumers to filter messages based on different business dimensions. In large-scale Internet businesses, there are also various timing event-triggered scenarios, the most typical of which is the order closing mechanism for transaction timeout. The scheduled messages of RocketMQ can easily meet such demands.
2. Consistency
RocketMQ also provides multiple key features in terms of consistency. The most representative is distributed transactional messages. RocketMQ is the first message queue to implement this feature. This feature can ensure that the upstream and downstream transactions are consistent with the order status. This solution has also become the de facto standard for asynchronous message consistency solutions and has been adopted by many Internet companies. Some companies have even ported it to a customized version of Kafka. In addition to distributed consistency, RocketMQ also provides the ordered message feature to meet the requirements of ordered consistency.
3. Stability
Stability is the cornerstone feature of transaction and financial scenarios, which is also the foundation of RocketMQ. In addition to HA with core services, RocketMQ also provides global high availability. Within Alibaba Cloud, RocketMQ supports advanced HA capabilities such as active zone-redundancy, active geo-redundancy, and central disaster recovery. Stability is not limited to the high availability of data and services. RocketMQ has built a full range of stability at the product level, such as message tracing, message backtracking, and message dead letter mechanisms.
4. High Performance
Due to the Double 11 limit traffic, the RocketMQ write message delay is 99.99% within 1 ms and 100% within 100 ms. RocketMQ uses a share-nothing distributed architecture and has the ability to expand infinitely in terms of throughput. It has supported Double 11 trillion message flood peaks for 10 consecutive years, providing millions of application instances with low latency Message Service.
In 2015, the first cloud message service of RocketMQ was launched on Alibaba Cloud, which started the large-scale cloud computing practice. RocketMQ is also the first open-source message queue in the industry to provide public cloud services. In large-scale cloud computing business scenarios, RocketMQ faces new opportunities and challenges.
1. Diversity
It no longer only serves the internal business of a certain company and is no longer limited to the Internet or financial enterprises. It needs to achieve the whole industry and the whole scene coverage.
2. Standardization
Self-developed message queues that only serve within an enterprise do not need to consider the standardization of protocols or APIs. However, for cloud message service, the service targets are external enterprise customers. Therefore, as a cloud message service, it needs to provide compatibility with industry de facto standard protocols, interfaces, and SDKs to ensure that customers can smoothly move to the cloud and simultaneously eliminate customers' concerns about technology binding.
3. The cloud-native concept is deeply rooted in people's mind
Message queues should better help customers achieve cloud-native application architecture, reduce costs and improve efficiency for business.
4. The rise of new technologies
These include IoT, 5G, edge computing, event-driven, and event streaming technologies. Facing these technical trends and diversified business requirements, RocketMQ has evolved itself to 5.0 versions.
In order to fully release the technical dividends of the cloud, RocketMQ 5.0 has evolved cloud-native in the technical architecture. This is a panoramic view of the cloud-native architecture of RocketMQ. The client and the server have been transformed to achieve higher elasticity, higher availability, and lower costs.
To meet the diverse needs of users in the cloud era, RocketMQ 5.0 expanded from the original Internet business message middleware to a message-event-stream hyper-integrated processing platform to unlock more comprehensive capabilities.
• In the message field, it fully embraces cloud-native technology, with better elastic architecture and high availability capabilities.
• In the event field, it supports the CloudEvent specification and a new event-centric product interface to help customers build a cross-business and cross-organization digital business ecosystem.
• In the stream field, the batch streaming storage feature is enhanced to significantly improve the data throughput. The logical queue capability is added to decouple logical resources and physical resources and seamlessly scale in streaming scenarios. The RSQLDB streaming database is added to provide real-time event stream processing and stream analysis capabilities.
Based on the end-cloud integration architecture, RocketMQ implements the capabilities of IoT message queue, expanding from the original connection application to the connection to IoT devices. At the same time, RocketMQ 5.0 also continues to maintain the principle of a simple architecture. It can build services with the lowest resource consumption and O&M costs and is suitable for edge computing.
In addition to the core capabilities, RocketMQ 5.0 is actively building an open source ecosystem. For one thing, it is the construction of the application architecture ecosystem, which includes both classic open-source projects and standard integration, such as JMS and AMQP, and cloud-native technology ecosystem integration, such as CloudEvents, Dapr, and Envoy. For another, RocketMQ will further develop the data architecture ecosystem from offline big data to real-time big data, and integrate big data ingestion, data storage, data processing, and data analysis components.
This course has three main points: the evolution of the message queue, the background of the birth of RocketMQ, and the driving force of its evolution.
The first part explores the evolution of message queues, which has gone through four stages: its early days, commercial dominance, the rise of open source, and the cloud era.
In the second part, we delved into RocketMQ's advantages, including its ability to handle large-scale and complex Internet services, its distributed consistency, its ability to meet the consistency requirements of transaction and financial scenarios, and its comprehensive design for stability. Additionally, it has high performance and can scale horizontally with the explosive growth of Internet services.
Finally, we learned about the driving force behind RocketMQ 5.0's evolution and the upgrade direction of the 5.0 version, based on the technology trends of the times. It is evolving towards cloud extension in terms of its technical architecture, and in the business field, it breaks through the original scenario of asynchronous decoupling of message queues and expands its features to include real-time data stream processing for the IoT.
In the next course, we will learn the analysis of RocketMQ 5.0 architecture.
Click here to go to the official website for more details.
Observability | Best Practices for Centralized Data Management of Multiple Prometheus Instances
RocketMQ 5.0: How Does the Cloud-native Architecture Support Diversified Scenarios?
206 posts | 12 followers
FollowAlibaba Cloud Native Community - November 23, 2022
Alibaba Cloud Community - December 9, 2021
Alibaba Cloud Native Community - January 5, 2023
Alibaba Cloud Native - June 12, 2024
Alibaba Developer - August 24, 2021
Alibaba Cloud Native Community - December 6, 2022
206 posts | 12 followers
Follow
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
Message Queue for Apache Kafka
A fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn More
Message Queue for RabbitMQ
A distributed, fully managed, and professional messaging service that features high throughput, low latency, and high scalability.
Learn More
ChatAPP
Reach global users more accurately and efficiently via IM Channel
Learn MoreMore Posts by Alibaba Cloud Native
