An Analysis of the Basic Concept of "Message-Driven, Event-Driven, and Streaming"

This article aims to provide a clear understanding of the recent high-frequency words "Message-Driven, Event-Driven, and Streaming" in messaging field.

By Kenmeng

This article aims to provide a clear understanding of the recent high-frequency words "Message-Driven, Event-Driven, and Streaming" in messaging field. Among them, Event-Driven Architecture (EDA) is one of the top ten technology trends according to Gartner, and EventBridge is one of the current focuses as the next-generation messaging middleware.

Alibaba Cloud RocketMQ 5.0 has been upgraded from message to the integration of message, event, and streaming. Based on this, Message-Driven, Event-Driven, and Streaming become high-frequency words in the messaging field recently. Since the three words are newly coined, many people may be confused.

Background

The translation of these three words:

Message-Driven: Message-driven communication
Event-Driven: Event-driven communication
Streaming: The streaming model

These three models are similar to asynchronous communication. The sender of the message will not wait for the consumer to respond to any data. Service decoupling is a common feature of the three models.

In the field of service communication, the following elements should also be considered in model selection:

Order: Whether events can be delivered in a specific order
Transaction: Whether producers or consumers can participate in distributed transactions
Persistence: How data is persisted and whether data can be replaced
Subscription Filtering: Whether the model supports subscription filtering based on Tags or other fields
At-least-once (delivered), At-most-once (delivered), and Exactly-once (delivered)

After the background introduction, let's look at each model in detail.

Message-Driven

The procedure in message-driven communication is when a Producer sends a message to a Consumer. The model is listed below:

Middleware is usually used in this model. Common Middleware includes RocketMQ, Kafka, and RabbitMQ. The function of the middleware is to cache the messages sent by the producer until the consumer is ready to receive them, thereby decoupling the systems on both ends.

The format of messages in the message-driven architecture is based on the needs of consumers; messaging can be one-to-one, many-to-many, one-to-many, or many-to-one.

A common example is the pushing of commodity orders. The upstream component is responsible for generating orders, and the downstream is responsible for receiving and processing orders. As such, the upstream does not need to care too much about the entire order lifecycle and focuses on generating orders quickly, so the performance of individual components can be improved.

The message-driven model provides light coupling between services (this coupling refers to Producer/Consumer SDK) and allows production and consumption to be extended according to demands.

Event-Driven

First of all, please note that the event-driven model is an improvement on the message-driven model. It imposes strict restrictions on the message size and format. This message-based restriction encapsulation is called an Event.

The producer publishes events in the event-driven model to indicate system changes, and any interested and authorized service can subscribe to these events and use them as triggers to initiate certain logic/storage/tasks.

Messaging can be one-to-one, many-to-one, one-to-many, or many-to-many in an event-driven model. Typically, multiple targets execute different events based on filtering conditions.

The format of the event in this model is formulated by the producer according to the event standard protocol. Thanks to the standard restrictions and encapsulation, the producer does not need to care about which systems are consuming the events it generates.

Events are not commands. Events do not tell consumers how to process messages. Their role is only to inform consumers that an event occurs at a certain time. As immutable data, events are important and have the same data value as messages. Normally, when an event occurs and is executed, it is often accompanied by another event.

The event-driven model provides minimal coupling between services and allows production and consumption services to be extended according to demands. It allows for the addition of various new components without affecting existing services.

There is a pertinent example of the event-driven model. We take a customer that purchases a product as an event to demonstrate the application of the model in the event scenario:

CRM (a customer relationship system) receives the customer's purchase information and can update his purchase record automatically.
EMR (an inventory management system) receives the customer's purchase information and dynamically adjusts and replenishes inventory in time.
After receiving the customer's purchase information, the express service will print the waybill and inform the express company to deliver it.

The event-driven model can be applied anywhere!

It is possible to filter, transform, schema, and search events in the productization of EventBridge, with the standardized encapsulation for messages. These capabilities also extend to more specific functions and related features for event-driven scenarios.

Streaming

Streaming is an ordered set of unbounded events or data. The execution operation is usually performed on a fixed event segment (e.g. 00:00 - 12:00) or a relative event (e.g. the past 12 hours).

Usually, the operation of a single event is implemented using the event itself, but the operations on the streaming are filtering, combining, splitting, mapping, etc.

The operations on the streaming can be stateless or stateful:

It is stateless for a single event, including filtering and mapping.
The time or location of the dependent messages in the streaming (e.g. offset, time) is stateful. The streaming processing must save some storage for consumed messages in stateful operations. The stateful operations include implementing Batch Size, Batch Window, and other operations on the data.

A simple example of streaming can be given here. For example, the logistics system will generate an event when an item passes through a logistics node, but if we want to know the complete logistics status event, there must be an aggregation of individual events of each logistics node. This aggregation event is a streaming event.

Kafka is the most typical streaming middleware. The location information of events is critical in streaming scenarios. Typically, the location information (e.g. offset) is managed by the consumer.

Standard Specifications for Event

After defining Event and Streaming, let's discuss the specifications.

The event specifications are to clarify the relationship between event producers and consumers. Currently, there are two major parts: AsyncAPI and CloudEvents.

AsyncAPI provides the corresponding Open API and Swagger based on the Event API.
CloudEvents focuses on processing the metadata of events.

The following introduces some related concepts of CloudEvents. The core function of CloudEvents is to define a set of metadata about events transmitted between different components and how the metadata should appear in the message body.

The main ideas of CloudEvents are listed below:

Make events more compliant with specifications
Reduce the difficulty of platform integration
Improve the portability of FaaS
Make source events traceable
Improve event relevance

Terms and Definitions

Occurrence refers to the logical occurrence of an event. An event occurs based on a certain situation.

Event indicates the data record of the event and the context. Routing can be determined based on the information in the event, but the event itself does not contain routing information.

Producer: This is the instance or component that creates the event.

Source: This is the context in which events occur, which can be composed of multiple producers.

Consumer: Receives and consumes events

Intermediary receives messages containing events and forwards them to the next receiver, similar to a router.

Context: Context metadata is encapsulated in context attributes to determine the relationship between events and other systems.

Data: It can also be called payload.

EventFormat: The event format, like JSON

Message: A message encapsulates the event and passes it from source to destination.

Protocol: It can be an industry standard (such as HTTP), an open-source protocol (such as Kafka), or a vendor protocol (such as AWS Kinesis).

Protocol Binding describes how to send and receive events through a given protocol and how to put events into messages.

Community

An Analysis of the Basic Concept of "Message-Driven, Event-Driven, and Streaming"

Background

Message-Driven

Event-Driven

Streaming

Standard Specifications for Event

Terms and Definitions

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

EventBridge

Media Solution

Function Compute

IDaaS