ApsaraMQ for RocketMQ:Message consumption retry

Triggers for message retry

• Consumption fails. This includes cases where the consumer returns a failure status or throws an unexpected exception.

• Message processing times out. This includes queuing timeouts in a PushConsumer.

Main behaviors of message retry

• Retry process state machine: Controls the state and transition logic of a message during the retry process.

• Retry interval: The time between the last consumption failure or timeout and when the message can be consumed again.

• Maximum number of retries: The maximum number of times a message can be retried.

Differences in retry policies

For more information about specific retry policies, see PushConsumer consumption retry policy and SimpleConsumer consumption retry policy.

Retry state machine

For example, the preceding figure shows the retry process for a message. Assume that the message stays in the Ready state for 5 s and the processing time is 6 s.

Each time the message is retried, its state changes from Ready to Inflight to WaitingRetry. The retry interval is the time between the last consumption failure or timeout and the time when the message can be consumed again. The actual time between two consumption attempts also includes the processing time and the duration in the Ready state. For example:

• The message enters the Ready state at 0 s for the first consumption attempt.

• Due to the consumer's processing speed, the message is not pulled for consumption until 5 s has passed. After 6 s of processing, an exception occurs, and the client returns a consumption failure.

• The message cannot be retried immediately. It must wait for the retry interval to elapse before it can be consumed again.

• At 21 s, the message returns to the Ready state.

• After another 5 s, the client starts to consume the message again.

Therefore, the actual interval between two consumption attempts is: Processing time + Retry interval + Duration in Ready state = 21 s.

Retry interval

• Unordered messages (non-ordered messages): The retry interval uses a tiered schedule. The following table lists the specific intervals.

  Retry attempt	Retry interval	Retry attempt	Retry interval
  1	10 seconds	9	7 minutes
  2	30 seconds	10	8 minutes
  3	1 minute	11	9 minutes
  4	2 minutes	12	10 minutes
  5	3 minutes	13	20 minutes
  6	4 minutes	14	30 minutes
  7	5 minutes	15	1 hour
  8	6 minutes	16	2 hours

  Note

  If the number of retries exceeds 16, the retry interval is 2 hours for each subsequent attempt.

• Ordered messages: The retry interval is fixed. For more information about the value, see Parameter limits.

Maximum number of retries

• Default: 16.

• Maximum: 1,000.

The maximum number of retries for a PushConsumer is controlled by the metadata of the consumer group. For more information about how to modify this value, see Modify the maximum number of retries.

For example, if the maximum number of retries is 3, the message can be delivered up to 4 times: 1 delivery of the original message and 3 retries.

Usage example

To trigger a message retry for a PushConsumer, you can return a consumption failure status code. The SDK also catches unexpected exceptions.

SimpleConsumer simpleConsumer = null;
        // Sample code: Use a PushConsumer to consume normal messages. If consumption fails, return an error to trigger a retry.
        MessageListener messageListener = new MessageListener() {
            @Override
            public ConsumeResult consume(MessageView messageView) {
                System.out.println(messageView);
                // Return a consumption failure to automatically trigger a retry until the maximum number of retries is reached.
                return ConsumeResult.FAILURE;
            }
        };
            

View consumption retry logs

Retries for ordered consumption by a PushConsumer occur on the consumer client. The server cannot retrieve detailed logs for consumption retries. If the delivery result of an ordered message in the message trace is "failed", check the consumer client logs for information such as the maximum number of retries and the consumer client.

For more information about the consumer client log path, see Log configurations.

You can search for the following keywords in the client logs to quickly locate content related to consumption failures:

Message listener raised an exception while consuming messages
Failed to consume fifo message finally, run out of attempt times

Retry state machine

Unlike the PushConsumer retry policy, the retry interval for a SimpleConsumer is pre-allocated. Each time a message is received, the consumer sets an invisibility duration parameter, InvisibleDuration, when it calls the API. This duration is the maximum processing time for the message. If a consumption failure triggers a retry, you do not need to set the next retry interval because the value of the invisibility duration parameter is reused.

Because the invisibility duration is pre-allocated, it may differ significantly from the actual message processing time in your business. You can use an API operation to modify the invisibility duration.

For example, if you set the maximum processing time to 20 ms but the message cannot be processed within that time, you can modify the message's invisibility duration to extend the processing time. This prevents the message from triggering the retry mechanism.

As shown in the following figure, the change to the invisibility duration takes effect immediately. The invisibility duration is recalculated from the moment the API is called.

Message retry interval

Message retry interval = Invisibility duration - Actual message processing time

The consumption retry interval for a SimpleConsumer is controlled by the message's invisibility duration. For example, if the invisibility duration is 30 ms and the message processing takes 10 ms before a failure response is returned, the next retry occurs after 20 ms. In this case, the message retry interval is 20 ms. If the message is not processed and no result is returned after 30 ms, the message times out and is retried immediately. In this case, the retry interval is 0 ms.

Maximum number of retries

• Default: 16.

• Maximum: 1,000.

The maximum number of retries for a SimpleConsumer is controlled by the metadata of the consumer group at creation. For more information about how to modify this value, see Modify the maximum number of retries.

For example, if the maximum number of retries is 3, the message can be delivered up to 4 times: 1 delivery of the original message and 3 retries.

Usage example

To trigger a message retry for a SimpleConsumer, you can wait for the message to time out.

 // Sample code: Use a SimpleConsumer to consume normal messages. To trigger a retry, simply wait for the message to time out, and the server will automatically retry.
        List<MessageView> messageViewList = null;
        try {
            messageViewList = simpleConsumer.receive(10, Duration.ofSeconds(30));
            messageViewList.forEach(messageView -> {
                System.out.println(messageView);
                // If processing fails and you want the server to retry, simply ignore the message. You can try to receive it again after its invisibility duration expires.
            });
        } catch (ClientException e) {
            // If the pull fails due to system throttling or other reasons, you need to initiate a new request to receive messages.
            e.printStackTrace();
        }

gRPC SDK

• Call the UpdateConsumerGroup OpenAPI operation.

• Modify in the console:

  You can perform this operation as follows:

  1. On the Instances page, click the name of the target instance.

  2. In the navigation pane on the left, click Groups. On the Groups page, click Create Group.

  

Remoting SDK

• Modify using a Remoting SDK parameter: Modify the value of the `maxReconsumeTimes` property of the consumer.

Use retries reasonably and avoid triggering them for needs such as rate limiting

As mentioned in the Scenarios section, message retry is suitable for scenarios where business processing fails and the current consumption failure is a low-probability event. It is not suitable for scenarios with persistent failures, such as rate limiting.

• Incorrect example:

  If the current consumption speed is too high and triggers rate limiting, return a consumption failure and wait for the next retry.

• Correct example:

  If the current consumption speed is too high and triggers rate limiting, delay receiving messages and consume them later.

How do I set the consumption timeout?

The consumption timeout is set on the consumer client. The following sections describe the parameter settings.

consumer.setConsumeTimeout(15); // The unit is minutes. The minimum value is 1 minute, and the maximum value is 180 minutes.