Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Implementation Practices of Message Queue for Apache RocketMQ 5.0

Implementation Practices of Message Queue for Apache RocketMQ 5.0

This article introduces the evolution of Message Queue for Apache RocketMQ 5.0 from the perspectives of large scalability, ease of use, observability, and SLA-based stability.

Preface

The Apache RocketMQ community revealed the product capabilities of Message Queue for Apache RocketMQ 5.0 at the RocketMQ Summit 2022 in July. The community also explained the technical positioning and development direction of Message Queue for Apache RocketMQ 5.0 for developers.

1

With rich experience in cloud computing, we have witnessed the continuous development of Message Queue for Apache RocketMQ over the past seven years. Now, Message Queue for Apache RocketMQ 5.0, the latest version of Message Queue for Apache RocketMQ, has officially been launched.

According to the community, as cloud-native and cloud adoption have become inevitable trends, Apache RocketMQ has updated its architecture and adapted some of its capabilities to meet the requirements of business developers. How can Message Queue for Apache RocketMQ 5.0 be implemented in production? This article focuses on the cloud-native transformation of messaging architectures and product capabilities. It also describes how Alibaba Cloud has adapted its products and technologies around Message Queue for Apache RocketMQ 5.0 and how they are upgrading products to better suit the technical and capability demands of the enterprise customers.

Development of Cloud-Native Messaging Services

First, let's take a look at how cloud-native messaging services will evolve.

As we venture into the future, cloud-native messaging products must make breakthroughs in the following aspects:

2

  • Large Scalability: The prime reason enterprises adopt cloud technologies is to lessen the burden on resource supply so they can focus on business integration and development. As an operator and maintainer of messaging services, we must provide the resource resilience businesses require in their course of development. In short, we need to be able to solve risks that may arise due to fluctuating traffic and find ways to help enterprises improve resource utilization.
  • Ease of Use: Ease of use is an important capability of integrated middleware. When we develop messaging services, we must optimize the API design, development process, configurations, and O&M to reduce the burden on users and help them avoid mistakes. We can open up the market to attract more users by lowering or removing the barriers for users to make use of the technology.
  • Observability: Observability is vital to all participants of messaging services. Service providers help system operators reduce O&M burden and empower users to troubleshoot issues by themselves by providing observation and diagnosis capabilities with clear boundaries and open standards.
  • High SLA-Based Stability: Since stability is a key capability of the production system, and messaging services are typically integrated in core transaction systems, the messaging system must specify the metrics that indicate the availability and reliability of the service, and users must design a troubleshooting mechanism and redundancy security mechanism based on a clear SLA.

In the following sections, let me briefly introduce how Message Queue for Apache RocketMQ 5.0 implements the preceding four aspects in practice.

Large Scalability: Provide Optimal Resources Based on Business Models

Messaging services are typically integrated in core systems and widely used in scenarios that feature large traffic fluctuations (such as transactions and payments). We usually see huge amounts of business traffic during special occasions (such as big sales promotions and flash sales).

In such cases, Message Queue for Apache RocketMQ 5.0 is an ideal choice. It can automatically adapt to business requirements by scaling resources. On the one hand, Message Queue for Apache RocketMQ 5.0 reserves resources for enterprises to deal with their basic business needs based on the cost optimization policy. On the other hand, it supports auto scaling to ensure business stability when traffic spikes occasionally. In this case, you are charged for additional resources that you use. This allows you to reserve minimal resources to ensure reliable services while ensuring business continuity during unexpected traffic spikes.

3

In addition to scalability, the messaging system is a stateful system that stores a large amount of valuable business data. When the number of API calls fluctuates, the storage also needs to scale up or down. In this case, you have to prevent data loss while saving storage costs. The scaling of conventional disk-based architectures can be problematic. For one, the capacity of local disks is limited. You can only add nodes to local disks if you want to scale up. This results in a waste of computing resources. For two, local disks cannot be dynamically scaled down. You can only reduce storage costs by isolating the business traffic. However, this operation is very complicated.

4

By comparison, Message Queue for Apache RocketMQ 5.0 stores messages in Serverless mode. You are charged for storage space based on actual usage. You only need to set a proper value for time-to-live (TTL) to ensure data integrity for long-term storage.

Ease of Use: Simplify Business Development for Easier Use

Ease of use is a requirement for system design. When we develop messaging services, we must optimize the API design, development process, configurations, and O&M to reduce the burden on users and help them avoid mistakes. For example, when consumers consume messages in Message Queue for Apache RocketMQ 4.X or Message Queue for Apache Kafka, they are often troubled by the load balancing policy. They have to pay attention to the number of queues and the number of consumers in the current topic because consumers implement load balancing and allocate tasks based on queues. If the capabilities for processing messages are different among consumers or the number of messages assigned to each consumer is different, some consumers will have long queues.

However, in a typical business integration scenario, the consumer client only needs to consume messages in a stateless messaging model, and consumers only need to focus on whether the messages are processed instead of the storage model and policy.

Based on this requirement, Message Queue for Apache RocketMQ 5.0 provides a new consumer model called simple consumers. This model supports the consumption, retry, and commitment of a single message.

5

Observability: Provide Automatic Diagnosis with Clear Boundaries and Open Standards

6

Personnel with experience in message queue O&M know that the messaging system decouples upstream production and downstream consumption. As a result, when a fault occurs, it is difficult to tell whether the fault is due to the messaging service or business logic.

The observability feature of Message Queue for Apache RocketMQ 5.0 provides solutions to the preceding problem using events, traces, and metrics. The following describes the content of the events, traces, and metrics in Message Queue for Apache RocketMQ 5.0:

  • Events cover O&M events on the broker (such as the configurations of downtime, restart, and changes) and change events on the client (such as the triggering of the subscription, subscription canceling, and online and offline events).
  • Traces cover the lifecycle of a message or trace. Traces show how a message is processed from production to storage to consumption. Based on the timeline, traces help capture all the participants in message processing and locate issues.
  • Metrics cover metrics related to observability and warning and help quantize various capabilities of the messaging system. Examples of metrics include TPS, throughput, traffic, storage space, failure rate, and success rate.

7

Message Queue for Apache RocketMQ has achieved a lot in terms of observability. It is the first of its kind to support comprehensive query of message traces. The latest version (5.0) supports reporting traces and metrics in the client and broker to a third party for storage using the standard OpenTelemetry protocol. Message Queue for Apache RocketMQ 5.0 can display and analyze data in a standard way with help from open-source technology (like Prometheus and Grafana).

SLA-Based Stability: Provide Valuable, Quantifiable, and Well-Defined Service Guarantees

Stability is a key capability of the production system, and messaging services are typically integrated in core transaction systems. Therefore, the stability of the messaging system directly impacts the stability of a business. We have to optimize O&M management and the design of the system architecture to ensure stability. We must quantize the boundaries and metrics of services. Only when we specify the metrics that are available and reliable for services can we design a troubleshooting mechanism and redundancy security mechanism.

8

Conventional O&M-based stability solutions are only applicable for basic scaling and metric monitoring. The conventional O&M-based stability guarantee cannot provide good quantitative services for various scenarios with complicated boundaries in messaging services (such as message accumulation, cold data reading, and broadcasting). Once these scenarios are triggered, the system fails.

By comparison, Message Queue for Apache RocketMQ 5.0 provides systematic stability through system design. This ensures quantitative services for scenarios (such as message accumulation and cold data reading) and helps determine message sending RT, end-to-end delay, and messaging throughput (TPS). Once these scenarios are triggered, Message Queue for Apache RocketMQ 5.0 can protect the system from failures in certain cases.

This article introduces the evolution of Message Queue for Apache RocketMQ 5.0 from the perspectives of large scalability, ease of use, observability, and SLA-based stability. This article also talks about how Message Queue for Apache RocketMQ 5.0 implements these four aspects in practice.

Currently, Message Queue for Apache RocketMQ 5.0 is available for commercial use. We have enhanced its features, scalability, ease of use, and O&M capabilities. Meanwhile, the prices of instances are up to 50% lower than those of the previous generation. We aim to help enterprises lower their costs and improve their efficiency in business development and integration. The Message Queue for Apache RocketMQ 5.0 instances support free scaling at a scale of 0 to 1 million TPS. They also support burst scaling and Serverless storage. In terms of observability, the Message Queue for Apache RocketMQ 5.0 instances support the integration of traces and metrics. In terms of ease of use, the Message Queue for Apache RocketMQ 5.0 instances support the next-gen lightweight native SDKs for multiple languages, making them more stable and easier to use.

RocketMQ Cloud Computing Computing Message Queue cloud native SLA Observability messaging services
0 1 0
Share on

Read previous post:

Cloud-Native Observability Suite: Building Ubiquitous Observability Infrastructure

Read next post:

Message Queue for Apache RocketMQ: Ordered Messages

Alibaba Cloud Native Community

508 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

508 posts | 48 followers

Related Products

  • ApsaraMQ for RocketMQ

    ApsaraMQ for RocketMQ is a distributed message queue service that supports reliable message-based asynchronous communication among microservices, distributed systems, and serverless applications.

    Learn More

  • Function Compute

    Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.

    Learn More

  • AliwareMQ for IoT

    A message service designed for IoT and mobile Internet (MI).

    Learn More

  • Elastic High Performance Computing Solution

    High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.

    Learn More

More Posts by Alibaba Cloud Native Community

See All