To prevent business exceptions caused by repeated message consumption, the Simple Message Queue (SMQ, formerly MNS) provides the consumption idempotence feature. This enables consumers to perform idempotent processing on the message based on the unique business identifier after receiving a message. This topic introduces the concept, scenarios, and processing method of message idempotence.
What is message idempotence?
In a message queue, a message may be repeatedly delivered to a consumer due to network exceptions or system failures. In this case, you can perform idempotent processing on the message to ensure that the business logic is executed only once. This helps minimize data errors or result inconsistencies caused by repeated message consumption.
For example, a consumer deducts the payment of an order based on a payment deduction message. The amount of the payment is USD 100. The message is repeatedly delivered to the consumer due to network exceptions or system failures. As a result, the message is repeatedly consumed. However, the payment is deducted only once, with only one deduction record of USD 100 for the order. In this example, message idempotence is implemented in the message consumption process, and the payment deduction meets business requirements.
Scenarios
When a system encounters network jitter, system faults, repeated message, and other anomalies, messages, especially in distributed systems, may be consumed repeatedly. In this case, message idempotence can be used to improve the reliability and consistency of the system. The possible reasons for repeated consumption of the same message are as follows:
Message consumption duration exceeds invisible duration: When a consumer A takes longer to process a message than the value of the InvisibleDuration parameter for the mesage, the message will be re-delivered to consumer B, leading to both consumers A and B processing the same message.
Network exceptions or jitters: After a message has been successfully processed by the consumer, network is disconnected. In this case, the request
DeleteMessageis not responded, and the message will be re-delivered to the consumer after network recovers.Broker or service consumer restart: After a message has been successfully processed by the consumer, the system may fail to respond to the
DeleteMessagerequest if you restart broker or service consumer. In this case, the message may be re-delivered to the consumer after the invisible duration.
Processing method
In a message queue, message IDs are usually used as idempotent keys for performing idempotent processing on messages. However, messages with different message IDs may contain the same content. To avoid conflicts or repetition, do not implement idempotent processing on messages based on message IDs. We recommend that you set an identifier as the idempotent key that uniquely identifies your business.
Unique identifier: Assign a unique identifier to each message. When a message is delivered to a consumer, the unique identifier of the message is recorded. This helps ensure that the message will not be consumed again by this consumer.
Database constraint: Set a unique constraint in the database. You cannot insert the data that was stored due to constraint violation, thus achieving idempotence.
Periodic check of business state: Consumers cannot consume a message before they check the current state of the mesaage. For example, consumers need to check whether the message exists or has been processed. If so, the message is directly returned without executing subsequent operations.
Merging operation: To achieve idempotence, you can merge some messages for ceratin operation. In this way, the fianl consumption result is the same as the initial consumption result.