Symptoms

A ApsaraMQ for Kafka instance that uses Apache Kafka V2.2.0 or later is used. The system does not delete the consumer offsets that are expired. The consumer offsets are still displayed on the Subscriptions tab of the details page of the topic in the ApsaraMQ for Kafka console.

Note The Consumer Offset Retention Period parameter specifies the maximum period of time that consumer offsets can be retained. The retention period of a consumer offset starts from the point in time when the consumer offset is committed. When the retention period of a consumer offset reaches the maximum period of time that is allowed, the system deletes the consumer offset. Each time a consumer offset is committed to a topic partition, the system resets the retention period. For more information, see Modify configurations for messages.

Causes

The group is active. In this case, the system does not delete consumer offsets of the group, regardless of whether the retention period of the consumer offsets exceeds the maximum retention period. This is a mechanism of Apache Kafka.Group

Note A group that contains an active consumer thread is active.

For example, you create a group named TestGroup to subscribe to Topic A, and consumer offsets are committed.Group Then, your business requirements change. You configure the TestGroup group to unsubscribe from Topic A and subscribe to Topic B. In this case, the TestGroup group is active. The system does not delete the consumer offsets for Topic A and Topic B.Group

If you configure the TestGroup group to unsubscribe from Topic A and do not configure the group to subscribe to Topic B, the TestGroup group is not active. In this case, the system deletes the consumer offsets whose retention period exceeds the maximum retention period.

If your business requires to immediately process messages and cannot wait the existing consumer offsets to expire and be deleted, you can create a new group in the same instance to subscribe to Topic B.Group If your business can wait until the consumer offsets expire and are deleted, you can use the TestGroup group to subscribe to Topic B after the consumer offsets for Topic A are deleted.

Solutions

To delete an expired consumer offset, perform the following steps:

  1. If your business requires the group to remain active, you can log on to the Message Queue for Apache Kafka console and manually reset the expired consumer offset for the required partitions.Group For more information, see Reset consumer offsets.
  2. If your business allows you to disconnect the group from Message Queue for Apache Kafka to make the group inactive, you can perform one of the following operations:Group
    • Disconnect the group from Message Queue for Apache Kafka to make the group inactive and make sure that the group remains inactive until the consumer offset expires. After the consumer offset expires, the system automatically deletes the consumer offset.
    • Migrate the consumer threads from the active group to another group. The consumer threads in the original group becomes inactive. Make sure that the group remains inactive until the consumer offset expires.Group After the consumer offset expires, the system automatically deletes the consumer offset.