ApsaraMQ for Kafka:Why are messages in different partitions consumed out of order or at different points in time with an obvious lag?

Problem description

On the Groups page in the ApsaraMQ for Kafka console, click the name of the group that you want to manage. On the Group Details page, click the Consumer Status tab and then click Consumer Details in the Actions column of the topic that you want to manage. In the panel that appears, if the value in the Maximum Offset column of a partition is larger than the values in the Maximum Offset column of other partitions, the partition receives messages from the producer earlier than other partitions.

Cause

Each consumer in the same group consumes messages in an evenly divided number of partitions. As a result, whether consumers in a group can evenly consume messages is determined by whether the number of partitions is divisible by the number of consumers.

• If the default number of partitions 12 is divisible by the number of consumers N, each consumer can evenly consume messages.

• If the default number of partitions 12 is not divisible by the number of consumers N, specific consumers consume more messages than other consumers.

  For example, 5 consumers and 12 partitions are available. In this case, two consumers each consume messages in three partitions, and the other three consumers each consume messages in two partitions. The consumption rate depends on the processing performance of the consumers. If all consumers have the same processing performance, the two consumers that each consume messages in three partitions consume messages more slowly than the three consumers that each consume messages in two partitions.

  Note

  The default number of partitions for a subscription or pay-as-you-go ApsaraMQ for Kafka instance is 12, and the default number of partitions for a serverless ApsaraMQ for Kafka instance is 3.

Solution

Make sure that the number of partitions is divisible by the number of consumers.