ApsaraMQ for Kafka:Why are the messages in different partitions consumed at different points in time with an obvious lag or why are the consumptions in different partitions out of order?

Problem description

On the Groups page of the ApsaraMQ for Kafka console, find the required consumer group and choose More > Consumption Status in the Actions column. On the Consumption Status tab of the Group Details page, all the topics to which the consumer group has subscribed and the values of the Messages Accumulated and Last Consumed At parameters of each topic are displayed. If the consumption time indicated for a partition is earlier than that for other partitions, this partition may receive messages from the producer earlier than other partitions.

Cause

Each consumer in the same Group consumes messages in an evenly divided number of partitions. Under this condition, whether messages are evenly consumed depends on whether the producer evenly sends messages to each partition.

• If the number (N) of consumers can be divided by 24 (the default number of partitions) without a remainder, each consumer consumes messages in N/24 partitions.
• If the number of consumers cannot be divided by 24 without a remainder, specific consumers may consume messages in one more partition than other consumers do.

  For example, five consumers and 24 partitions are available. In this case, four consumers each consumes messages in five partitions, and the remaining consumer consumes messages in four partitions. The speed of consumption depends on the processing performance of the consumers. If all consumers have the same processing performance, the four consumers that each consumes messages in five partitions may consume messages more slowly than the consumer that consumes messages in four partitions.

Solution

Make sure that the number of consumers can be divided by 24 without a remainder.