Message Queue for Apache RocketMQ: Ordered Messages

By Shao Shu

Overview

This article introduces the scenarios where messaging services are integrated. This article also describes the working mechanism, use cases, and best practices of the ordered messages feature in Message Queue for Apache RocketMQ.

Preface

Since its development about a decade ago, Message Queue for Apache RocketMQ has been proven to be a stable solution for a wide variety of businesses. Now, it is widely implemented in Alibaba Group's internal processes and used by over tens of thousands of Alibaba Cloud enterprise users. As a messaging solution with financial-grade reliability, Message Queue for Apache RocketMQ has always focused on building its asynchronous communication capacity in business integration. This article introduces the scenarios where messaging services are integrated. This article also describes the working mechanism, use cases, and best practices of the ordered messages feature in Message Queue for Apache RocketMQ.

Overview

Ordered messages are a type of message provided by Message Queue for Apache RocketMQ. These types of messages are delivered and consumed in a strict first-in-first-out (FIFO) order in a topic. The earliest delivered message is consumed first, and the latest delivered message is consumed last. However, this only applies to messages in the same message group. Message Queue for Apache RocketMQ does not guarantee the order of messages across message groups. In actual practice, we must pay attention to the order of message sending and the order of message consumption.

Working Mechanism

A frequently asked question from Message Queue for Apache RocketMQ users: If ordered messages are an enhanced version of normal messages, why don't we only use ordered messages? I will answer this question in the following section by comparing normal messages and ordered messages.

Ordered Message Sending

It can be difficult to ensure that messages are globally ordered in a distributed environment. For example, if Producer A and Producer B independently send Message A and Message B to a Message Queue for Apache RocketMQ broker, we cannot verify the order in which messages were sent. This is one of the drawbacks of a distributed environment. Therefore, the messaging systems in the industry typically focus on the order of messages with the same attribute. This attribute is known as a message group. In the following figure, Producer A sends Message A1 and Message A2 from Message Group A and Message B1 and Message B2 from Message Group B. Producer B sends Message C1 and Message C2 from Message Group C.

We can construct a single-thread scenario where different producers manage different message groups to preserve the order in which messages in the same message group are sent, and the producers send ordered messages in a synchronous manner. Synchronous message sending ensures a message is only sent after the preceding message is sent to the client. This is an advantage over asynchronous message sending or multi-thread message sending.

Even though the underlying working mechanism of ordered messages and normal messages seem the same, the synchronous message sending mode is used for ordered messages to preserve the order of messages. Compared with the asynchronous message sending mode used for normal messages, synchronous message sending reduces the maximum throughput of messages significantly.

Ordered Message Consumption

No restrictions are enforced on how normal messages are consumed. Consumers can consume messages in an asynchronous and concurrent manner. However, for ordered messages, consumers can only consume one message in a specific message group at a specific time. They cannot consume the next message in the same message group until the preceding message is consumed or sent to the dead-letter queue. This mechanism is put in place to preserve the order of the messages. As a result, the business logic that the user uses to process the message will affect the consumption rate. Message accumulation may occur in extreme cases where there are too many messages in a message group. Of course, this only happens for individual message groups. Messages across groups can be consumed in parallel. So, the message order mentioned in this topic is a partial order.

Summary

We use message groups to manage messages. Messages are sent and consumed in a parallel manner across message groups. The message group feature enables ordered messages to implement multiple-queue storage, horizontal partitioning, and concurrent consumption. Different from ordered messages, each message group of normal messages only contain aone message.

Let's look at the question mentioned at the beginning of this article:

If ordered messages are an enhanced version of normal messages, why don't we only use ordered messages?

After the preceding explanation, I guess you have already realized that the advantages of ordered messages come at a cost.

The following table compares ordered messages and normal messages.

Best Practices

Properly Configure a Message Group

Message groups are complex, and properly setting up a message group may be a challenge to some users. Let’s take an e-commerce platform as an example. The e-commerce platform configures seller IDs as the message group. However, some large-scale sellers may produce a large number of orders. These orders may accumulate because the downstream consumption capability is limited. In this case, the e-commerce platform should configure order IDs as the message group. Meanwhile, we should take the underlying business logic into account to identify the messages whose order need to be preserved. For this scenario, only messages that share the same order ID should have their message order preserved. The best practice for configuring a message group is to make sure the lifecycle of the message group is short, and the number of messages in different message groups is evenly distributed.

Synchronous Message Sending and Message Retries

We must use synchronous message sending and message retries to preserve message order.

Idempotent Consumer

If exceptions occur in the messaging system, some messages may be processed more than once. In this case, you can implement idempotent consumer logic to handle duplicate messages.

Use Cases

Consider a scenario where a verification code is required for user registration, and the user ID is used as the message group. The messages sent by the same user are published and consumed in FIFO order.
In an e-commerce scenario, the order ID is used as the message group. Then, messages about the creation, payment, delivery, and refund status of the same order are published and consumed in FIFO order.

Example

Message Sending

The following sample code provides an example of how to send a message in a message group using the synchronous message sending mode:

public class ProducerFifoMessageExample {
    private static final Logger LOGGER = LoggerFactory.getLogger(ProducerFifoMessageExample.class);

    private ProducerFifoMessageExample() {
    }

    public static void main(String[] args) throws ClientException, IOException {
        final ClientServiceProvider provider = ClientServiceProvider.loadService();

        // Credential provider is optional for client configuration.
        String accessKey = "yourAccessKey";
        String secretKey = "yourSecretKey";
        SessionCredentialsProvider sessionCredentialsProvider =
            new StaticSessionCredentialsProvider(accessKey, secretKey);

        String endpoints = "foobar.com:8080";
        ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
            .setEndpoints(endpoints)
            .setCredentialProvider(sessionCredentialsProvider)
            .build();
        String topic = "yourFifoTopic";
        final Producer producer = provider.newProducerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the topic name(s), which is optional. It makes producer could prefetch the topic route before 
            // message publishing.
            .setTopics(topic)
            // May throw {@link ClientException} if the producer is not initialized.
            .build();
        // Define your message body.
        byte[] body = "This is a FIFO message for Apache RocketMQ".getBytes(StandardCharsets.UTF_8);
        String tag = "yourMessageTagA";
        final Message message = provider.newMessageBuilder()
            // Set topic for the current message.
            .setTopic(topic)
            // Message secondary classifier of message besides topic.
            .setTag(tag)
            // Key(s) of the message, another way to mark message besides message id.
            .setKeys("yourMessageKey-1ff69ada8e0e")
            // Message group decides the message delivery order.
            .setMessageGroup("youMessageGroup0")
            .setBody(body)
            .build();
        try {
            final SendReceipt sendReceipt = producer.send(message);
            LOGGER.info("Send message successfully, messageId={}", sendReceipt.getMessageId());
        } catch (Throwable t) {
            LOGGER.error("Failed to send message", t);
        }
        // Close the producer when you don't need it anymore.
        producer.close();
    }
}

Message Consumption

The following sample code provides an example of how to consume a message:

public class SimpleConsumerExample {
    private static final Logger LOGGER = LoggerFactory.getLogger(SimpleConsumerExample.class);

    private SimpleConsumerExample() {
    }

    public static void main(String[] args) throws ClientException, IOException {
        final ClientServiceProvider provider = ClientServiceProvider.loadService();

        // Credential provider is optional for client configuration.
        String accessKey = "yourAccessKey";
        String secretKey = "yourSecretKey";
        SessionCredentialsProvider sessionCredentialsProvider =
            new StaticSessionCredentialsProvider(accessKey, secretKey);

        String endpoints = "foobar.com:8080";
        ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
            .setEndpoints(endpoints)
            .setCredentialProvider(sessionCredentialsProvider)
            .build();
        String consumerGroup = "yourConsumerGroup";
        Duration awaitDuration = Duration.ofSeconds(30);
        String tag = "yourMessageTagA";
        String topic = "yourTopic";
        FilterExpression filterExpression = new FilterExpression(tag, FilterExpressionType.TAG);
        SimpleConsumer consumer = provider.newSimpleConsumerBuilder()
            .setClientConfiguration(clientConfiguration)
            // Set the consumer group name.
            .setConsumerGroup(consumerGroup)
            // set await duration for long-polling.
            .setAwaitDuration(awaitDuration)
            // Set the subscription for the consumer.
            .setSubscriptionExpressions(Collections.singletonMap(topic, filterExpression))
            .build();
        // Max message num for each long polling.
        int maxMessageNum = 16;
        // Set message invisible duration after it is received.
        Duration invisibleDuration = Duration.ofSeconds(5);
        final List<MessageView> messages = consumer.receive(maxMessageNum, invisibleDuration);
        for (MessageView message : messages) {
            try {
                consumer.ack(message);
            } catch (Throwable t) {
                LOGGER.error("Failed to acknowledge message, messageId={}", message.getMessageId(), t);
            }
        }
        // Close the simple consumer when you don't need it anymore.
        consumer.close();
    }
}

After introducing ordered messages in Message Queue for Apache RocketMQ, I hope you can have a deeper understanding of the working mechanism and application of ordered messages. I also hope this solution can help solve your problems in business in a more effective manner.

Community

Message Queue for Apache RocketMQ: Ordered Messages

Overview

Preface

Overview

Working Mechanism

Ordered Message Sending

Ordered Message Consumption

Summary

Best Practices

Properly Configure a Message Group

Synchronous Message Sending and Message Retries

Idempotent Consumer

Use Cases

Example

Message Sending

Message Consumption

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

ApsaraMQ for RocketMQ

Function Compute

Elastic High Performance Computing Solution

Quick Starts