ApsaraMQ for RocketMQ:Message idempotence

What is message idempotence?

When a consumer repeatedly consumes a message, the result of repeated consumption is the same as that of one consumption, and the repeated consumption does not affect the business system in a negative manner. In this case, message idempotence is implemented.

For example, a consumer deducts the payment of an order based on a payment deduction message. The amount of the payment is USD 100. The message is repeatedly delivered to the consumer due to network issues such as unstable network connections. As a result, the message is repeatedly consumed. However, the payment is deducted only once, and only one deduction record of USD 100 is generated for the order. In this example, message idempotence is implemented in the message consumption process, and the payment deduction meets business requirements.

Scenarios

In Internet applications, especially in the case of unstable network connections, ApsaraMQ for RocketMQ messages may be repeatedly consumed. If the repeated consumption of messages affects your business, you can implement idempotent processing on the messages.

Messages can be repeatedly consumed in the following scenarios:

• A producer repeatedly sends a message to a Message Queue for RocketMQ broker.

  If a transient network connection occurs or the producer breaks down after a message is sent to and persisted in the ApsaraMQ for RocketMQ broker, the broker fails to respond to the producer. In this case, the producer determines that the broker does not receive the message and sends the message again. As a result, the consumer receives two messages that have the same content and different message IDs.

• An ApsaraMQ for RocketMQ broker repeatedly delivers a message to a consumer.

  A message is delivered to a consumer and is processed by the consumer. When the consumer sends a response to the ApsaraMQ for RocketMQ broker, a transient network connection occurs. In this case, the ApsaraMQ for RocketMQ broker does not know whether the message is consumed. To ensure that the message is consumed at least once, the broker delivers the message again after the network recovers. As a result, the consumer receives two messages that have the same content and the same message ID.

• A message is repeatedly delivered to a consumer due to load balancing. The factors that trigger load balancing include but are not limited to network jitters, broker restarts, and consumer application restarts.

  Load balancing is triggered if you restart or scale the ApsaraMQ for RocketMQ broker or client. During load balancing, a consumer may receive repeated messages.

Implement message idempotence

Messages with different message IDs may contain the same message content. We recommend that you do not implement idempotent processing based on message IDs to avoid conflicts. We recommend that you use unique business identifiers to implement idempotent processing. You can use message keys to specify the business identifiers.

For example, in a payment scenario, you can specify the order ID as the message key and implement idempotent processing based on the specified message key. The following sample code provides an example on how to specify a unique business-specific key for a message:

Message message = new Message();
message.setKey("ORDERID_100");
SendResult sendResult = producer.send(message);           

When the consumer receives the message, the consumer can perform idempotent processing based on the message key.

consumer.subscribe("ons_test", "*", new MessageListener() {
    public Action consume(Message message, ConsumeContext context) {
        String key = message.getKey()
        // Perform idempotent processing based on the message key that uniquely identifies your business. 
    }
});