The development of message middleware has spanned over 30 years, from the emergence of the first generation of open-source message queues to the explosive growth of PC Internet, mobile Internet, and now IoT, cloud computing, and cloud-native technologies.
As digital transformation deepens, customers often encounter cross-scenario applications when using message technology, such as processing IoT messages and microservice messages simultaneously, and performing application integration, data integration, and real-time analysis. This requires enterprises to maintain multiple message systems, resulting in higher resource costs and learning costs.
In 2022, RocketMQ 5.0 was officially released, featuring a more cloud-native architecture that covers more business scenarios compared to RocketMQ 4.0. To master the latest version of RocketMQ, you need a more systematic and in-depth understanding.
Today, Qingshan Lin, who is in charge of Alibaba Cloud's messaging product line and an Apache RocketMQ PMC Member, will provide an in-depth analysis of RocketMQ 5.0's core principles and share best practices in different scenarios.
Today's course will focus on the event-driven aspects of RocketMQ 5.0. Event-driven is a classic concept, and through this course, you'll learn how event-driven in the cloud era differs from previous implementations.
The first part will cover the concept of event-driven from a technological philosophy perspective. The second part will introduce EventBridge, a new sub-product launched by RocketMQ 5.0 for event-driven architecture tailored to the cloud era. Finally, we'll explore several specific cases to help you understand event-driven solutions in the cloud era.
First, let's learn what event-driven is. From a definition perspective, event-driven is essentially a software design pattern that minimizes the coupling between different modules and systems.
Here's a typical event-driven architecture diagram. The event producer sends events to EventBroker, which then routes the events to the corresponding consumers for event processing. Event processing can be flexibly extended, and event consumers can be added or removed at any time, transparent to the event producer.
Why is event-driven a classic design pattern? Decades ago, various event-driven technologies emerged. For example, in desktop client-end programming frameworks, clicking a button could trigger an onclick event, and developers wrote business logic to respond to these events. In programming languages, event-driven code patterns are often used, such as callback and handler functions. In the era of distributed systems, communication and collaboration between systems will also be event-driven.
Have you noticed that this figure is very similar to the message application decoupling figure of RocketMQ before? Yes, both message publishing and subscription and event production and consumption are for code decoupling and system decoupling. Message queues are technical implementations, while most EventBroker implementations are based on message queues. Event-driven is more like an architectural concept.
From a technical perspective, message queues can be compared to RPC, with one representing synchronous communication and the other providing asynchronous communication. A message queue doesn't specify the content of the message; it only contains binary content. In fact, from a technical implementation standpoint, the core technology required for Event-Driven Architecture (EDA) is message queue technology. The main difference between event-driven and message-driven is that an event is a special kind of message. Only when a message has certain characteristics can it be called an event.
Let's take an example and look at the diagram on the left. Messages are like an abstract class with various subclasses, the most important of which are Command and Event. For instance, sending an Open message to traffic lights is a kind of Command. The traffic lights accept this Command and turn on the green light. After that, the lights send out a message that they have turned green, which is an Event.
There are four main characteristics of events:
First, events are immutable. An event indicates that something has happened and has become a fact.
Second, events have a time concept, and the sending of events is orderly for the same entity. For example, traffic lights send green, yellow, and red events in order.
Third, events are unexpected. This is why the EDA architecture can achieve maximum decoupling. The event producer doesn't care who the event consumer is or how they consume the event.
Fourth, since event-driven architecture is completely decoupled and has no expectations about how downstream components consume events, events should include as much detailed information as possible, allowing downstream consumers to extract what they need. For example, a traffic light event contains multiple fields, including its source, type, theme, which light is changed, and a unique ID for easy tracking, as well as the event time and content.
As a classic architecture model, event-driven has become a focus again in the cloud era. There are two main reasons for this:
Firstly, cloud-native technologies, such as microservices, have driven the adoption of event-driven architecture. By adopting a microservices architecture, digital enterprises can organize into smaller business units and teams that collaborate efficiently in a "high cohesion, low coupling" manner. However, this model also brings new problems, such as increased latency and reduced availability. An event-driven microservice system can improve the resilience of microservices, reduce latency, and achieve more complete decoupling.
Another representative cloud-native technology is the serverless architecture paradigm, which is also event-driven. Currently, the main trigger sources of serverless products, such as Function Compute of Alibaba Cloud or Lambda of AWS, are various types of events. For example, cloud product events, user business events, and cloud product O&M events can trigger functions to run consumption logic. The large-scale use of event-driven architecture can help digital enterprises unleash the technical dividends of serverless cloud computing.
IoT is also an important driving force for event-driven architecture. Many IoT applications are event-driven, such as sensor reporting events, temperature change events, and address location change events. On-cloud applications subscribe to these events to trigger corresponding workflow.
After the large-scale digital transformation of the entire industry, cross-business and cross-organization business cooperation will move from offline to online. In the digital economy era, the scale of the digital business ecosystem will continue to expand, and cross-organization business collaboration needs to be completely decoupled. The natural asynchronous and decoupling characteristics of EDA can solve these problems.
Next, we'll move on to the second part and learn about RocketMQ 5.0's EventBridge. Before trying to understand the technical implementation of this system, let's take a look at EventBridge's abstraction of event-driven generic capabilities, from which we can also learn about EventBridge's domain model.
Let's take a look at this picture from left to right. On the far left is the event source. Since this event is intended to be consumed across platforms, we want to adopt the industry standard as the event format. At the same time, events may be consumed across organizations, so we need a unified event center to register these different event sources to this event center. For consumers, it's like an event store where they can choose events that interest them. When an event consumer starts to write consumption logic, they also need to have a clear understanding of the format of the event. They need to know what the content of the event is, what the fields are, and what the meanings are, to write the correct consumption business logic.
Therefore, EventBridge also provides a schema center. With this schema center, consumers can know the format of events at a glance without having to communicate with the initiator of the event source. The overall efficiency is greatly improved. Looking further back, you can see the event consumption step. There are many types of event consumers, and different consumers pay attention to different event types, so EventBridge needs to provide rich filtering rules. Even if multiple consumers are interested in the same event, they may only need part of the event content. EventBridge also provides the ability to transform events.
This is the abstraction of the event-driven capabilities of RocketMQ 5.0.
In the era of cloud computing and large-scale digital transformation, we emphasize that event-driven architectures often span different organizations and different platforms. Therefore, the event-driven architecture needs a unified event standard. In the EventBridge product, we have adopted the CloudEvents standard from the CNCF Foundation, which is the de facto standard for industry events. This standard is designed to simplify event declaration and improve the interoperability of events across services and platforms.
CloudEvents brings many benefits:
First, it provides a specification that enables cross-organizational and cross-platform event integration with a common language to accelerate more event integration. Further, specifications can also accelerate the integration of cross-service and cross-platform events.
Second, with the popularity of serverless architecture, major cloud vendors provide Function Compute services. With the CloudEvents specification, users can use Function Compute without vendor binding.
Third, webhook is a common integration mode. With the CloudEvents specification as a unified format, webhooks of different systems can achieve better interoperability.
Finally, such a unified specification is actually conducive to the development of event-driven software infrastructure and facilities, such as cross-service event tracing analysis.
The following figure shows the new version of EventBridge launched by RocketMQ to handle EDA scenarios.
Its core technologies are all based on RocketMQ, but the product interface is abstracted for event-driven business, with the core domain objects changed from messages to CloudEvents. This builds an event-driven digital ecosystem based on a unified event standard. The event sources are diverse, including cloud product events, SaaS platform events, application custom events, and general webhooks. Of course, the event targets are also diversified. Events are routed to different consumers through the event rules engine. Typical consumers include Function Compute, storage systems, message notifications such as DingTalk SMS, and general webhooks. This completely decoupled architecture of event-driven is more suitable for building hybrid cloud and multi-cloud digital systems.
To improve the efficiency of event-driven R&D, EventBridge also supports schema features, interprets and previews event information, and even automatically generates code. This allows developers to integrate events in a low-code or zero-code way.
Another important feature of EventBridge is the event rules engine. Different event consumers are interested in different events. Therefore, we provide seven event filtering modes, including prefix matching, suffix matching, exception matching, and numeric value matching. You can perform various complex combinatorial logic filters to push only events that are of interest to consumers.
Of course, even if they all care about the same event, different consumers will have different information concerns within the event. To improve the efficiency of event consumption, we also provide four types of event converters, which can push only the event fields that consumers care about. You can also transform events into custom templates to meet more flexible business demands.
As a sub-project of RocketMQ, EventBridge also provides complete observability capabilities. EventBridge can query the event list according to the event time and type. Each event generates a unique ID. Users can accurately locate the event content, occurrence time, corresponding event rules, and downstream consumption status based on the unique ID, and accurately troubleshoot problems.
The following are some cases of using EventBridge.
The first case applies to companies that use massive cloud products. Customer C is a technology company focused on smart consumer terminals. It wants to collect all cloud events in its account for subsequent analysis or troubleshooting. EventBridge on the public cloud aggregates events from all cloud services. By using EventBridge, the customer can collect all events for custom business processing. It can also configure event rules, filter abnormal events, and push them to the monitoring system or DingTalk so that the customer can notice them timely.
The second case is the integration of SaaS events. Now with the prosperity of the entire cloud computing ecosystem, many enterprises not only use IaaS and PaaS products of public cloud but also use SaaS products of the third party, such as various ERP, CRM, and other systems. Based on the integration capability of HTTP and webhook of EventBridge standard, it can seamlessly connect the third party SaaS system as the event source. The enterprise can collect all SaaS events that it cares about and facilitate subsequent management, such as application forms, employment forms, reimbursement forms, orders, etc.
The third case is SaaS platform integration. Take DingTalk as an example. DingTalk is a typical SaaS platform with 4000 + ecosystem partners, including ISV ecosystem partners, hardware ecosystem partners, service providers, consulting ecosystem and delivery ecosystem partners, etc. EventBridge is used to connect the PaaS ecosystem of the public cloud with the SaaS ecosystem of DingTalk. EventBridge is used to manage the overall event lifecycle and push the event lifecycle to downstream ISV receivers in the form of webhooks. For example, DingTalk's official event sources include video conferences, schedules, address books, approval processes, DingTalk drive, Yida, etc. Enterprises and SaaS vendors can make full use of these official application events to build enterprise-level application systems, and can also deeply integrate DingTalk official data streams with other systems.
This course introduces the new significance of EDA in the cloud era. Its resurgence in the cloud era is mainly driven by technologies such as IoT and cloud-native technologies. Additionally, commercial driving forces are also being increasingly adopted as digital business ecosystems thrive. Subsequently, for event-driven scenarios in the cloud era, RocketMQ 5.0 has launched a sub-product that is characterized by embracing industry standards, enabling it to link events across platforms and organizations. It provides a powerful rules engine to flexibly connect upstream and downstream events. At the same time, it also provides schema capabilities, which improve the entire event-driven user experience and R&D productivity. Finally, we list several typical event-driven cases in the cloud era, such as how to manage cloud product events in a unified manner after users have fully connected to the cloud; how to use events from multiple SaaS platforms to build your own business system; and, as a SaaS platform itself, how to build a platform ecosystem based on EventBridge open standard events.
The In-depth Analysis of RocketMQ 5.0 series comes to a conclusion at this point, and we look forward to meeting you next time.
Click here to go to the official website for more details.
RocketMQ 5.0 IoT Messaging: What Message Technology Does IoT Need?
208 posts | 12 followers
FollowAlibaba Cloud Native Community - November 23, 2022
Alibaba Cloud Native - June 6, 2024
Alibaba Cloud Native - June 12, 2024
Alibaba Cloud Native - June 7, 2024
Alibaba Cloud Native Community - January 5, 2023
Alibaba Cloud Native Community - November 20, 2023
208 posts | 12 followers
Follow
ApsaraMQ for RocketMQ
ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.
Learn More
AliwareMQ for IoT
A message service designed for IoT and mobile Internet (MI).
Learn More
Message Queue for RabbitMQ
A distributed, fully managed, and professional messaging service that features high throughput, low latency, and high scalability.
Learn More
Message Queue for Apache Kafka
A fully-managed Apache Kafka service to help you quickly build data pipelines for your big data analytics.
Learn MoreMore Posts by Alibaba Cloud Native
