By Qinjun
This article describes how Alibaba Cloud applies the serverless architecture to message queues to effectively reduce operational costs and provides fast response and flexible communication capabilities for IoT devices based on the features of the cloud-native environment.
With the rapid development of smart appliances, industrial Internet, and Internet of Vehicles, the demand for messaging for IoT (Internet of Things) devices is growing at an unprecedented rate. In this context, efficient and reliable messaging standards become crucial. As a widely recognized protocol in a new generation of IoT scenarios, the MQTT protocol is gradually becoming an industry standard.
We will introduce the MQTT core design built on RocketMQ. This article focuses on how RocketMQ adapts to these changes and promotes the efficiency and scalability of message processing in IoT scenarios by optimizing the storage and computing architecture, push model, and server architecture design to implement the MQTT protocol.
In addition, Alibaba Cloud MQTT continues to iterate and innovate based on RocketMQ-MQTT. Alibaba Cloud is one of the major contributors and users of open-source RocketMQ-MQTT. Faced with the challenges in device communication fluctuations, this article describes how Alibaba Cloud applies the serverless architecture to message queues to effectively reduce operational costs and provides fast response and flexible communication capabilities for IoT devices based on the features of the cloud-native environment.
Furthermore, we will discuss the practice of integrating MQTT in the cloud ecosystem and introduce the data ecosystem integration solution based on unified storage to demonstrate its powerful technical capabilities and flexible data forwarding capabilities.
Comparison of message scenarios:
| Item | Classic Message Scenarios | IoT Message Scenarios |
|---|---|---|
| Hardware Resources | Servers: containers, virtual machines, and physical machines | Micro devices with extremely limited computing and storage resources |
| Network Environment Conditions | Internal IDC has high bandwidth and stability. | Public network, complex environment, poor connectivity, unstable network |
| Message Client Scale | The size of the application server generally does not exceed 100,000. | Terminal scale: hundreds of millions |
| Production Mode | Each message client generates a large amount of data. | Each message client generates a small amount of data. |
| Consumption Mode | Each message has up to ten recipients. | Each message may have millions of recipients. The message recipient is a device. |
IoT technology, as a shining star in the field of contemporary technology, has achieved widespread consensus regarding its rapid development. The amount of IoT data is growing at an astonishing annual growth rate of about 28%, indicating that more than 90% of real-time data in the future data ecosystem will originate from IoT. This trend has significantly changed the landscape of data processing, pushing real-time stream data processing to a new stage centered around IoT data.
The rise of edge computing is a proactive response to this transformation. It is expected that in the future, up to 75% of data processing tasks will be completed at the edge, away from traditional data centers or the cloud. Given the massive amount of IoT data, relying on the cloud for all data processing tasks is both costly and difficult to meet low latency requirements. Therefore, the effective use of edge computing resources to preliminarily process data in place and upload the extracted information to the cloud has become a key strategy to improve efficiency and optimize user experience.
In this context, the messaging mechanism in IoT scenarios increasingly shows its core values:
• Bridging: The message system acts as a neural network in the IoT world. It seamlessly connects devices to devices and devices to cloud-based applications, which builds a cloud-edge-end integrated application framework to ensure instant and efficient information exchange.
• Data processing engine: Faced with the continuous surge of IoT data, event stream storage and stream computing technologies based on message queues (MQ) become the key to unlocking the potential of real-time data analysis. The messaging mechanism cannot only capture and store data streams in real time but also support computing operations at the moment the data is generated, providing a powerful data processing infrastructure for IoT applications and contributing to instant data insight and decision response.
In short, messaging technology is both the bond of the IoT architecture and the core impetus for data flow and intelligent decision-making. The depth and breadth of its application in the IoT field are expanding with the dual drive of technology iteration and market demand.
Meanwhile, there are many differences between traditional and IoT message scenarios, including the following features:
1) Differences in hardware resources
Traditional message scenarios rely on high-performance and high-reliability service clusters with abundant computing resources. The client deployment environments are mostly containers, virtual machines, or even physical servers, emphasizing centralized computing capabilities. In contrast, clients in IoT message scenarios are directly embedded into micro devices at the edge of the network, such as sensors and smart appliances. These devices hold extremely limited computing and storage resources, raising high requirements for energy efficiency.
2) Challenges to the network environment
In a classic internal data center (IDC) environment, message processing enjoys stable network conditions and controllable bandwidth and latency metrics. However, since the IoT environment is extended to public networks, it faces complex and changeable network conditions. Unstable connections become common conditions, especially in remote areas or areas with weak network coverage, posing higher challenges to the robustness and efficiency of message transmission.
3) Magnitude increase of client scale
The average daily message volume processed by traditional message systems is usually maintained at millions, which is suitable for message distribution with high concentration. The rise of IoT has led to an explosive growth in the number of devices, involving hundreds of millions of terminal nodes, which puts forward new requirements for the scalability of the system and the efficiency of message routing.
4) Evolution of production and consumption modes
Traditional message scenarios tend to adopt a centralized synchronous production mode, which emphasizes the consistency and orderly processing of messages. The IoT message generation exhibits a decentralized characteristic, with each device independently generating a small number of messages based on its perceived environment, which presents new considerations for the design of message collection and the integration mechanism. In terms of consumption mode, IoT scenarios often involve large-scale broadcast or multicast, and a single message can be sent to millions of receivers, requiring the system to have efficient broadcast capabilities and flexible subscription management mechanisms.
Converged architecture diagram
We can see that the IoT messaging technology greatly varies from the classic message design. Next, let's take a look at the features of MQTT, a converged architecture based on RocketMQ, designed to address IoT message scenarios.
1) Integration with the MQTT protocol to adapt to the characteristics of the IoT environment
By integrating with RocketMQ-MQTT, RocketMQ 5.0 closely conforms to the MQTT protocol standard widely used in the IoT field. This protocol is optimized for low power consumption and unstable network conditions and stands out with its lightweight features and rich feature set. It supports different levels of messaging assurance, meeting diverse requirements from "at most once" to "only once". The domain model of the protocol is highly coordinated with the core components of RocketMQ, promoting the natural integration of messages, topics, and publish-subscribe models, laying a solid foundation for establishing a seamless messaging system from the device to the cloud.
2) Flexible storage and computing decoupling architecture
To address the requirements for high-concurrency connections and large-scale data processing in IoT scenarios, RocketMQ 5.0 adopts an architecture that separates storage and computing. As a core storage component, RocketMQ Broker ensures data persistence and reliability. MQTT-related logical operations are implemented at a dedicated proxy layer, which optimizes the management of a large number of connections and complex subscription relationships and also enhances real-time message push capabilities. This architecture allows for dynamic adjustments of the proxy layer resources based on the service load and smoothly copes with the increase in the number of device connections by adding proxy nodes, reflecting the good elasticity and expansion potential of the system.
3) Integrated architecture to promote end-cloud data collaboration
RocketMQ-MQTT facilitates efficient data sharing between IoT devices and cloud applications through its integrated architecture design. Based on the unified message storage strategy, each message is only stored once in the system for consumption at both ends, reducing data redundancy and improving the efficiency of data circulation. In addition, as a storage hub for data streams, RocketMQ is combined with stream computing technology to facilitate the real-time analysis of massive data generated by IoT and accelerate the process of data value discovery.
The first issue to be addressed is the storage model of IoT messages. In the publish-subscribe business model, there are two common storage models, write amplification and read amplification. We will analyze the two models in turn.
Write amplification model
Read amplification model
In this model, each consumer has its exclusive message queue, and each message needs to be copied and distributed to the queues of all target consumers. Consumers only pay attention to and process the messages in their own queue. Take the three-level topic, Topic/subTopic/test, as an example. If the topic attracts a large number of client subscriptions, adopting a "one queue per consumer" strategy means maintaining a separate message copy for each client or wildcard subscription that matches the subscription rules. This would result in the message storage requirements increasing linearly with the growing number of subscribers.
This model works well in some traditional message scenarios, such as applications that follow the AMQP protocol, because it ensures the isolation and reliability of messaging. However, in IoT scenarios, especially when a single message needs to be consumed by millions of devices, the "write amplification" strategy will result in serious storage resource consumption issues, which will quickly become an unbearable burden.
Considering the special needs of IoT scenarios, it is not feasible to directly apply the traditional "write amplification" model. To address this issue, RocketMQ-MQTT adopts a more efficient and flexible storage strategy, aiming to reduce storage redundancy and improve overall system scalability and resource utilization:
In the "read amplification" model, each message is actually stored once. To support efficient retrieval of wildcard subscriptions, the system creates additional index information during the message storage stage, that is, the consume queue. For wildcard subscriptions like /Topic/subTopic/*, the system generates a corresponding index in each matching wildcard queue so that consumers who subscribe to different wildcard topics or specific topics can find and consume messages through these shared storage entities. Although it seems to increase the complexity of "reading", the volume of each consume queue as an index is much smaller than that of the original message, which significantly reduces the overall storage cost and improves the efficiency of message retrieval and distribution.
Atomic distribution diagram
To this end, we have designed a Topic queue model for multi-dimensional distribution. As shown in the preceding figure, messages can come from various access scenarios, such as MQ/AMQP on the server and MQTT on the client, but only one copy will be written to the commitlog, and then distributed to queue indexes (ConsumerQueue) in multiple demand scenarios. For example, in the server scenario (MQ/AMQP), traditional server consumption can be performed based on the first-level Topic queue. In the client MQTT scenario, messages can be consumed based on MQTT multi-level Topics and wildcard subscriptions. Such a queue model can support both server and client scenarios for access and message sending and receiving, achieving the goal of integration.
To implement this model, RocketMQ relies on two key technical features:
• Light Message Queue: This feature allows a message to be flexibly written to multiple topic queues, ensuring that the message can efficiently adapt to various complex subscription modes, including but not limited to wildcard subscriptions. It provides the necessary flexibility and efficiency foundation for the implementation of the read amplification model.
• Million-queue Capability: RocketMQ integrates RocksDB, a high-performance key-value storage engine, and makes full use of its advantages in sequential writing to manage millions of queues. Particularly, through customized configuration, the Write-Ahead Log (WAL) within RocksDB is removed, which further optimizes the storage efficiency. RocksDB provides a stable and efficient storage solution for the consume queue and guarantees that the system can maintain high-performance index processing capabilities even with extremely massive queues.
Implementation of the light message queue
By using the read amplification model, combined with the light message queue feature and the underlying technical support of millions of queues in RocketMQ, we effectively solve the challenges of message storage and distribution in the IoT environment and achieve dual optimization of storage costs and system performance. This design reduces storage space usage and speeds up message retrieval through a highly optimized indexing mechanism, providing an economical and efficient solution for message communication of large-scale IoT devices.
RocketMQ-MQTT push model
After introducing the underlying queue storage model, we will discuss the implementation mechanisms for matching search and reliable delivery. In the traditional RocketMQ, consumers directly initiate long polling requests through the client to accurately retrieve queue messages of the corresponding topic. However, in MQTT scenarios, the long polling mode is not effective due to the large number of clients and complex subscription relationships, making the consumption process more complex. To address this issue, we adopt a push-pull combination model.
The core of this model is that the terminal connects to the proxy node through the MQTT protocol and messages can come from multiple scenarios such as MQ, AMQP, and MQTT. After the message is stored in the topic queue, the notification logic module will monitor the arrival of a new message in real time, generate a message event (that is, the topic name of the message), and push it to the gateway node. The gateway node performs internal matching based on the subscription status of the connected terminal, identifies the terminal capable of receiving this message, and then triggers a pull request to read the message from the storage layer and push it to the terminal.
In this process, a key point is how the notification module determines the messages that are of interest to the terminal and gateway nodes that are interested in such messages. This is actually a core matching search issue. There are two common solutions: the first is simple event broadcasting, and the second is to store online subscription relationships in a centralized manner (such as the lookup module in the figure), then perform a matching search, and push the message to the specific terminal.
Although the event broadcasting mechanism has limited scalability, it still performs well because the amount of data we push is relatively small, only the topic name. Additionally, message events of the same topic can be merged into one event, which is the default method we adopt in the production environment. On the other hand, it is also a common practice to store online subscription relationships in a centralized manner in RDS or Redis, but this needs to ensure the real-time nature of data because the matching search process may affect the latency of the overall real-time message chain.
In this model, a cache module is also designed to avoid repeated read requests to the storage layer from various terminals when broadcasting a large number of messages, thereby improving the efficiency of the overall system.
As cloud-native technology continues to develop, modern message-oriented middleware is gradually based on container orchestration. How to achieve serverless architecture and second-level elastic management has become an important research topic.
As one of the major contributors and users of open-source RocketMQ-MQTT, Alibaba Cloud has many optimization methods and practical experience in MQTT scaling design. We will introduce the scaling design ideas of Alibaba Cloud to show how it implements efficient and flexible MQTT message-oriented middleware:
1) Abstract the network connection layer
Alibaba Cloud MQTT adopts a Sidecar-like mode to separate the network connection layer from the core business logic and handles network connections with the Rust language. Compared with Java, Rust has significant advantages in memory consumption and startup speed, especially when handling large-scale MQTT connections, which can effectively reduce memory usage.
2) Second-level scale-out
Each pod has a low resource request setting, while some pods are reserved to run Rust processes exclusively. When scale-out is required, the system can quickly start the MQTT proxy process, saving the time of pod creation and resource mounting, thereby significantly improving the response speed.
3) Predictive scaling and monitoring
Using white-box metrics such as the number of connections, TPS, memory, and CPU, as well as black-box metrics such as RT, Alibaba Cloud MQTT formulates a reasonable scale-out strategy based on metric interaction rules. This enables the system to predict load changes in advance and initiate pod scale-out to ensure long-term smooth operation.
Based on the above design ideas, Alibaba Cloud can build an efficient and flexible MQTT implementation solution based on RocketMQ, making full use of the performance advantages brought by Rust while maintaining the stability and scalability of the system. This innovative design will significantly improve the user experience in practical applications and optimize the overall performance of the system.
With the advancement of the new four modernizations including electrification, intelligence, connectivity, and sharing in the automotive industry, major automakers are gradually building the IoV system with intelligent driving and intelligent connectivity as the core. This new generation of the IoV system places higher requirements on the platform architecture for the underlying message collection, transmission, and processing. In this section, we will introduce the practical architecture of Alibaba Cloud MQTT in the Internet of Vehicles and its application value.
In the architecture diagram, we can see common IoV devices, including on-board terminals, road test units, and mobile phone systems. These devices ensure connection and data transmission security. The functions of the vehicle devices cover new services such as vehicle data reporting, POI delivery, file push, configuration delivery, and message push. These operations generate a large number of message topics, which require more secure and stable access and transmission to achieve reliable message subscription and publishing. The road device emphasizes the safe access of the roadside RSU and supports the collection and transmission of messages and the real-time update of map data.
The access devices support multiple protocols, including TCP, x509, TLS, WSS, WS, OpenAPI, and AMQP to meet the flexible requirements of different application scenarios. The multi-protocol support ensures seamless connection and efficient communication between devices.
In terms of forwarding ecology, in the IoT scenario, various devices continue to generate a large amount of data, and the business party needs to analyze and process these data in depth. By using RocketMQ as the storage layer, the system can store only one copy of a message and support the shared consumption of IoT devices and cloud applications. The stream storage feature of RocketMQ enables the stream computing engine to seamlessly analyze IoT data in real time, providing timely support for critical decisions.
With Alibaba Cloud EventBridge, the information generated by MQTT IoT devices can be smoothly forwarded to other middleware or data processing platforms, such as Kafka, AMQP, FC, and Flink, for in-depth data analysis and processing. EventBridge is a serverless event bus service that is provided by Alibaba Cloud. You can connect Alibaba Cloud services, custom applications, and software as a service (SaaS) applications to EventBridge in a standardized and centralized manner. You can use EventBridge to route events among the preceding applications based on the standardized CloudEvents 1.0 protocol. You can also use EventBridge to build loosely coupled and distributed event-driven architectures. This flexible data forwarding capability not only improves the processing speed but also lays the foundation for the innovation and development of forthcoming intelligent applications.
The preceding architecture clearly shows the best practices of Alibaba Cloud MQTT in the field of Internet of Vehicles, providing reliable technical support for future intelligent transportation.
Against the backdrop of the vigorous development of IoT, the continuous evolution of messaging technology has become an important cornerstone supporting fields such as smart appliances, industrial Internet, and the Internet of Vehicles. Through the deep integration of RocketMQ and MQTT protocols, we effectively solve the demand for efficient and reliable message transmission in the IoT era and provide flexible solutions for device communication.
Alibaba Cloud is actively exploring this field, continuously advancing the technical iteration and innovation of MQTT by introducing the serverless architecture. Such a design dynamically adjusts resource allocation in the face of high-concurrency connections and massive data to reduce costs and improve response speed, ensuring the efficiency of real-time data processing.
Currently, the community has made significant progress in promoting the MQTT 5.0 protocol. New protocol features such as richer error codes, more flexible connection options, and will messages, retain messages, and shared subscription functions will further enhance the flexibility and reliability of the system. Meanwhile, we are committed to achieving faster elastic scaling capabilities to respond promptly in the face of burst traffic, improving system availability and flexibility.
With the continuous improvement of the IoT ecosystem, the value of messaging technology is becoming more prominent in increasingly complex message scenarios. We believe that in the future, through the continuous optimization of the message architecture, we can promote deeper intelligent applications and lay a solid foundation for the future where everything is connected. Let's expect MQTT to demonstrate its infinite possibilities in IoT technology scenarios, while also continuing to explore message-oriented middleware that ensures security, stability, and efficiency.
206 posts | 12 followers
FollowAlibaba Cloud Native - June 12, 2024
Aliware - November 10, 2020
Alibaba Cloud Native Community - November 20, 2023
Alibaba Cloud Native - June 7, 2024
Alibaba Cloud Native Community - July 12, 2022
Alibaba Cloud Native Community - January 5, 2023
206 posts | 12 followers
Follow
IoT Platform
Provides secure and reliable communication between devices and the IoT Platform which allows you to manage a large number of devices on a single IoT Platform.
Learn More
AliwareMQ for IoT
A message service designed for IoT and mobile Internet (MI).
Learn More
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
IoT Solution
A cloud solution for smart technology providers to quickly build stable, cost-efficient, and reliable ubiquitous platforms
Learn MoreMore Posts by Alibaba Cloud Native
