A Reinterpretation of RocketMQ Commit Log Storage Protocol

By Wei Li, Apache RocketMQ Committer, RocketMQ Python Client Project Owner, Apache Doris Contributor, and TencentDB Development Engineer

Recently, it occurred to me that lots of software and hardware are not what they are by design. There is a root reason for their designs: to meet the demand at that time and in that scene. Once I realize that, I feel that I'm talking to the designers. I understand their ideas and learn their methods, and I resonate with them: live and learn.

1. Have a Think

Is the consumer queue offset continuous, and why?
Is the commit log offset continuous, and why?
Are bytes in Java files stored in big-endian order or little-endian order by default, and why?

2. Real Distribution of the Commit Log

Bearing the preceding questions in mind, let's review the distribution of commit logs.

We can find the following commit log files under the root directory in which the configuration files of the broker are stored.

Real distribution of data files of the broker

It can be seen that there are multiple real stored files, each of which is named after a string resembling a number and 1 GB in size.

It can be learned from the source code that the actual abstract model is as follows:

Abstract model of commit log file distribution

As shown in the preceding figure:

• Commit log is the name of a type of file. There are multiple commit log files, each of which is called a commit log file.

As shown in the preceding figure, there are a total of T commit log files. These files are stored by creation time.

• Each commit log file stores messages, and messages are stored in the order in which they are written. Messages are always written in the latest file that is created and in one thread at a time.

As shown in the preceding figure, the number 1 indicates the first message in the first file. Such is the case for numbers 2, 3, 4, and others. It can be seen that the 1234th message is the last in the first commit log file, and the 1235th message is the first in the second commit log file.

Note 1: The actual storage space occupied by all messages in each commit log file is at most 1 GB. Think about the reason for it.

Note 2: Each time you write a commit log, RocketMQ locks the log. See https://github.com/apache/rocketmq/blob/7676cd9366a3297925deabcf27bb590e34648645/store/src/main/java/org/apache/rocketmq/store/CommitLog.java#L676-L722 for the code snippets.

Append a lock

In the commit log file, many messages are stored according to the established protocol. What is the specific protocol and how do you know that?

3. Commit Log Storage Protocol

Let’s take a look at the following source code: https://github.com/apache/rocketmq/blob/rocketmq-all-4.9.3/store/src/main/java/org/apache/rocketmq/store/CommitLog.java#L1547-L1587

Commit log storage protocol

The protocol fields are put together in the following figure.

Commit log storage protocol from what I understand

Note: The message protocol number in the preceding figure is different from that in the code. The latter only indicates the order of the protocol, while the storage protocol in physical files will be more detailed.

Here are a few questions that need to be clarified.

• The binary protocol uses byte orders, which are often referred to as big-endian and little-endian orders. These orders will not be described in detail here. If you are interested, google it or ask ChatGPT about it. They will provide more detailed answers.

• In Java, each data type occupies a certain number of bytes. A byte occupies 1 byte, an int 4, a short 2, and a long 8.

• The host does not directly convert the IP:Port value as a string to a byte array. Instead, it encodes each number as a byte. This will be explained in the following Golang code section.

• In the encoding of extended information, two invisible characters are used as segmentation. Therefore, the extended fields as key-value pairs cannot contain these invisible characters. What are these two characters?

You have learned about the protocol, but how to prove that your physical files are written according to this protocol?

4. Decode RocketMQ Commit Log by Using Golang

RocketMQ is written in Java. According to the storage protocol described in the preceding section, I built a tool in Golang to decode commit logs and consumer queues.

Code repository: https://github.com/rmq-plus-plus/rocketmq-decoder

Currently, this tool supports the following two features.

• Specify the commit log offset, directly parse the messages in the commit logs, and print the results.

• Specify the consumer offset, parse the consumer queues to obtain the commit log offset, parse the commit log based on the commit log offset, and print the results.

In Golang, no code relies on RocketMQ to decode. Instead, it relies purely on the protocol.

Import in Golang

Here is an example of parsing a commit log offset in Golang. In Java, a commit log offset is a long data type and occupies 8 bytes.

In Golang, a commit log offset is obtained by reading the data of 8 bytes in length and decoding it to int 64 in big-endian order.

Golang-demo

Here is the result of running the demo.

Read consumer-queue-commit-log

5. Answer the Preceding Questions

The following is my understanding of the preceding questions.

5.1 Is the Consumer Queue Offset continuous, and why?

The consumer queue offset is continuous.

The Consumer Queue Offset refers to the subscript index in each queue, which is continuous. Consumers also take advantage of this continuity to avoid consumer offset holes.

Each index occupies 20 bytes. The following figure illustrates the structure of the index.

Index Structure of consumer-queue

The physical offset in the figure refers to the Commit Log Offset.

5.2 Is the Commit Log Offset continuous, and why?

The Commit Log Offset is not continuous.

The Commit Log Offset refers to the byte offset of each message in all commit log files. The size of each message is uncertain, so the Commit Log Offset, or the byte offset, varies.

Additionally, it's worth noting that the absolute value of the difference between every two offsets is the total bytes of the previous message.

Moreover, the previous figure "Abstract model of commit log file distribution" contains a misunderstanding. The size of each small square is different.

5.3 Are bytes in Java files stored in big-endian order or little-endian order by default, and why?

Bytes in Java files are stored in big-endian order. Byte order includes the data storage order and the network transmission order. By default, Java adopts the big-endian format, which aligns with the order in which data is transmitted in a network, making encoding and decoding more convenient.

The first byte of the data message of each network transmission layer indicates the protocol used to transmit the following data. This allows the data receiver to first analyze the protocol according to the byte order when receiving the data, and then decode the subsequent byte sequence based on the protocol, aligning with the way humans think about and solve problems.

The above is my understanding. Please feel free to leave comments if you have any questions.

Note: There may be differences between different versions of RocketMQ. This article only discusses RocketMQ 4.9.3.

Community

A Reinterpretation of RocketMQ Commit Log Storage Protocol

1. Have a Think

2. Real Distribution of the Commit Log

3. Commit Log Storage Protocol

4. Decode RocketMQ Commit Log by Using Golang

5. Answer the Preceding Questions

5.1 Is the Consumer Queue Offset continuous, and why?

5.2 Is the Commit Log Offset continuous, and why?

5.3 Are bytes in Java files stored in big-endian order or little-endian order by default, and why?

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

ApsaraMQ for RocketMQ

Storage Capacity Unit

Hybrid Cloud Storage

Hybrid Cloud Distributed Storage