If messages in ApsaraMQ for RocketMQ still fail to be consumed after the maximum number of retries is reached, the messages become dead-letter messages and are forwarded to dead-letter queues. After you troubleshoot the issue, you can resend the dead-letter messages for consumption. If you cannot process the dead-letter messages, you can export and save the messages to prevent them from being deleted after expiration.
Background information
When a message fails to be consumed, ApsaraMQ for RocketMQ automatically retries the message. If the message still fails to be consumed after the maximum number of retries is reached, the consumer cannot consume the message as expected. In this case, ApsaraMQ for RocketMQ sends the message to a special queue that is specified for the consumer. For more information, see Message retry.
In ApsaraMQ for RocketMQ, messages that cannot be consumed as expected are known as dead-letter messages. Queues that are used to store dead-letter messages are known as dead-letter queues.
Characteristics
Dead-letter messages have the following characteristics:
Dead-letter messages can no longer be consumed as expected by consumers.
By default, the validity period for dead-letter messages is 3 days, which is the same as the validity period for normal messages. Expired dead-letter messages are automatically deleted. We recommend that you handle dead-letter messages within 3 days after they are generated.
Dead-letter queues have the following characteristics:
Each dead-letter queue corresponds to a consumer group rather than a consumer instance.
If no dead-letter messages are generated in a consumer group, ApsaraMQ for RocketMQ does not create a dead-letter queue for the consumer group.
A dead-letter queue contains all dead-letter messages that are generated in the corresponding group, regardless of to which topics the messages belong.
You can query, export, and resend dead-letter messages in the ApsaraMQ for RocketMQ console.
Methods to query dead-letter messages
The following table describes the methods that you can use to query dead-letter messages in ApsaraMQ for RocketMQ.
Method | Condition | Type | Description |
By group ID | Group ID + Time range | Range match | You can specify a group ID and a time range, and then retrieve all the messages that meet the conditions in batches. This query method allows you to query a large number of messages by using fuzzy match. |
By message ID | Group ID + Message ID | Exact match | You can specify a group ID and a message ID, and then locate a message by using exact match. |
Query dead-letter messages
Log on to the ApsaraMQ for RocketMQ console. In the left-side navigation pane, click Instances.
In the top navigation bar, select a region, such as China (Hangzhou). On the Instances page, click the name of the instance that you want to manage.
In the left-side navigation pane, click Dead-letter Queues. On the Dead-letter Queues page, use one of the following methods to query dead-letter messages:
Query by Group ID
You can specify a group ID and a time range during which dead messages are generated. Then, query all messages that meet the conditions.
NoteThe generation time of a dead-letter message refers to the time when a message is sent to the dead-letter queue after the maximum number of retries specified for the message is reached.
Query by Message ID
If you query messages by message ID, exact match is used. You can specify a group ID and a message ID to find the message that you want to query by using exact match.
Export dead-letter messages
If you cannot process dead-letter messages, you can export the messages in the ApsaraMQ for RocketMQ console to prevent them from being deleted after expiration.
The ApsaraMQ for RocketMQ console allows you to export a single dead-letter message or multiple dead-letter messages at a time. The exported files are in the CSV format.
The following table describes the fields of an exported dead-letter message.
Field | Description |
instanceId | The ID of the ApsaraMQ for RocketMQ instance. |
topic | The topic to which the message belongs. |
msgId | The message ID. |
bornHost | The URL of the producer that produced the message. |
bornTimestamp | The timestamp that indicates when the message was produced. |
storeTimestamp | The timestamp that indicates when the message was stored to the ApsaraMQ for RocketMQ broker. |
reconsumeTimes | The number of consumption failures. |
properties | The message attributes in the JSON format. |
body | The message body that is encoded in Base64. |
bodyCRC | The message body Cyclic Redundancy Check (CRC). |
Export a single dead-letter message
In the ApsaraMQ for RocketMQ console, find the dead-letter message that you queried and click More in the Actions column. Then, select Export Message from the drop-down list to export the dead-letter message.
Export multiple dead-letter messages at a time
In the ApsaraMQ for RocketMQ console, select the dead-letter messages that you queried by group ID and click Batch Export Messages to export the dead-letter messages.
Resend dead-letter messages
Only messages that cannot be consumed as expected are forwarded to a dead-letter queue. Dead-letter messages require special processing. After you troubleshoot the issues, you can resend the dead-letter messages for consumption in the ApsaraMQ for RocketMQ console.
If an ordered message is sent to a dead-letter queue, you cannot resend the message. You must export the message from the dead-letter queue and process the message.
After a dead-letter message is re-sent for consumption, the message is stored in the dead-letter queue for a specific period of time. The system automatically deletes the message after the validity period elapses.
Resend a single dead-letter message
In the ApsaraMQ for RocketMQ console, find the dead-letter message that you queried and click More in the Actions column. Then, select Resend from the drop-down list to resend the dead-letter message.
Resend multiple dead-letter messages at a time
In the ApsaraMQ for RocketMQ console, select the dead-letter messages that you queried by group ID and click Batch Resend Messages to resend the dead-letter messages.
References
For information about the working mechanism of message retry, see Message retry.