Symptoms

When you use Alibaba Cloud ApsaraMQ for Kafka, you receive alerts about message accumulation for a consumer group even after the consumer group is deleted.

Causes

The alerts about message accumulation are reported due to the following reasons:
  • Apache Kafka does not provide API operations that allow you to delete consumer groups and consumer offsets for Message Queue for Apache Kafka instances whose broker version is earlier than 2.2.0. After you delete a consumer group for these Message Queue for Apache Kafka instances, the consumer group is logically deleted only from the console. The information such as the consumer offsets of the consumer group is not deleted from the broker. Alerts about message accumulation are handled based on consumer offsets. Therefore, you still receive alerts about message accumulation.
  • After a consumer group is deleted, the consumer threads in the consumer group do not stop or no rebalance operation is triggered. In this case, the consumer threads continue to consume messages. As a result, you still receive alerts about message accumulation.

Solutions

If you do not want to receive alerts about message accumulation for a consumer group after it is deleted, perform the following operations:
  • Check the broker version. Then, perform the following operations based on the broker version:
    • For versions 2.2.0 and later, perform the following operations:
      • If the consumer group does not contain an active consumer thread, directly delete the consumer group. A consumer thread is active if it uses the subscribe method to consume messages. If you still receive an alert after the consumer group is deleted, check whether a consumer thread is submitting consumer offsets.
      • If an active consumer thread exists in the consumer group, you can reset the consumer offsets of the partitions that do not need to be counted for message accumulation to 0. This way, the system stops collecting information about message accumulation of the partitions. For more information, see Reset consumer offsets.
    • For versions earlier than V2.2.0, perform one of the following operations based on your business requirements:
      • Wait for consumer offsets to expire. After the maximum retention period of a consumer offset elapses, the consumer offset expires. In versions earlier than V2.2.0, Apache Kafka does not provide API operations that allow you to delete consumer offsets. Therefore, you cannot delete consumer offsets. If a consumer offset is not updated after the specified maximum retention period elapses, the consumer offset is automatically deleted. For information about how to configure the maximum retention period for consumer offsets, see Modify configurations for messages.
      • In specific scenarios, you cannot wait for consumer offsets to expire. In this case, make sure that the consumer group exists, and reset the consumer offsets of the partitions that do not need to be counted for message accumulation to 0 in the console. This way, the system stops collecting information about message accumulation of the partitions. For more information, see Reset consumer offsets.
      • If no active consumer thread exists in the consumer group, you can upgrade the broker version to 2.2.0 or later, recreate the consumer group, and then delete the consumer group. For information about how to upgrade the broker version, see Upgrade the versions of instances.
    Note In version 2.2.0 or later, if a consumer group contains an active consumer thread, a consumer offset in the consumer group is not deleted even if the maximum retention period elapses. For more information, see Why are expired consumer offsets retained?
  • If the issue persists, you can disable monitoring and alerting for message accumulation. For information about how to disable alert rules for message accumulation, see Monitoring and alerting.