If a message in ApsaraMQ for RocketMQ fails to be consumed, message retry is triggered. If the message still fails to be consumed after the specified maximum number of retries is reached, the message becomes a dead-letter message. ApsaraMQ for RocketMQ allows you to specify a topic to save dead-letter messages for subsequent business recovery or troubleshooting. This topic describes the common scenarios, policies, limits, configuration methods, and usage notes of dead-letter messages.
Common scenarios
Scenario in which dead-letter messages are correctly processed
After a message fails to be retried, you can save the message to a dead-letter topic and create a consumer group to consume the message. This helps you troubleshoot issues.
Scenario in which dead-letter messages are incorrectly processed
If a dead-letter message is forwarded in multiple systems and then dumped back to the original topic, the message is retried in a loop. This can cause a message avalanche.
Dead-letter policies
In what conditions does a message become a dead-letter message?
If a message fails to be consumed after the maximum number of retries is reached, the message becomes a dead-letter message.
Rules for saving dead-letter messages
By default, ApsaraMQ for RocketMQ does not save dead-letter messages. After a message becomes a dead-letter message, the message is discarded.
You can enable the dead-letter message saving feature in the ApsaraMQ for RocketMQ console. After you enable this feature, dead-letter messages are saved to a specific topic. This topic is known as a dead-letter topic. For more information, see Enable the dead-letter message saving feature.
After a message is saved to a dead-letter topic, the following rules apply:
A new message ID is generated for the message.
Custom configurations, such as custom attributes and the message body, remain unchanged.
The retention period of a dead-letter message starts from the point in time when the message is saved to the dead-letter topic. For example, a message is delivered to the broker at 13∶00∶00 and is sent as a dead-letter message to a dead-letter topic two hours later at 15:00:00. The retention period of the dead-letter message starts from 15∶00∶00.
Limits
You can specify only topics of the normal message type or ordered message type as dead-letter topics.
You cannot use the topic in which the dead-letter message is produced as the dead-letter topic. Otherwise, a message avalanche can occur. If the system detects that the dead-letter topic is the same as the original topic, the message is discarded.
You can save dead-letter messages from different topics to the same dead-letter topic.
If you delete the consumer group to which a dead-letter topic belongs, the dead-letter topic is not deleted.
If the topic that you want to delete is associated with a dead-letter policy, you must disassociate the topic from the policy before you delete the topic.
Enable the dead-letter message saving feature
You can specify whether to save dead-letter messages in the ApsaraMQ for RocketMQ console.
Operations:
On the Instances page, click the name of the instance that you want to manage.
In the left-side navigation pane, click Groups. On the Groups page, click Create Group.
Dead-letter message metrics
Description
Type | Metric |
rocketmq_send_to_dlq_messages: the number of new dead-letter messages per minute. | |
|
How to use metrics
ApsaraMQ for RocketMQ allows you to configure alert rules for dead-letter messages. This helps you detect exceptions at the earliest opportunity and troubleshoot issues based on the metric data that is displayed on the dashboard.
Scenario 1: Check the number of new dead-letter messages per minute
You can view the rocketmq_send_to_dlq_messages metric on the dashboard. You can also add the SendDLQMessageCountPerGid metric as a monitoring item in CloudMonitor.
Scenario 2: Check the number of unprocessed dead-letter messages
You can view the Consumer lag metric of the specified dead-letter topic on the dashboard. You can also add the ConsumerLagPerGidTopic metric as a monitoring item in CloudMonitor.
Usage notes
How does a consumer obtain the information about an original topic?
Solution 1: Map the name of the dead-letter topic to the name of the original topic when you configure a dead-letter topic.
For example, if the original topic is named as testTopic, name the dead-letter topic as DLQ-testTopic.
Solution 2: Include the information about the original topic in a custom attribute of the message. Sample code:
messageBuilder.addProperty("originalTopic","testTopic")
Separately process dead-letter messages
Dead-letter messages are messages that still fail to be consumed after retries are performed in the normal business workflow. To prevent business interruptions, you must separately process dead-letter messages.
Do not use the original topic of a message as the dead-letter topic. Otherwise, the dead-letter message is dumped to the original topic and retried in a loop. This can cause a message avalanche.
Do not consume dead-letter messages by using consumer groups that consume other messages.