The cause may be that the version of the ApsaraMQ for Kafka client is outdated, or the consumer does not have a separate thread to send heartbeat messages.
Symptoms
Rebalances frequently occur on my consumer client when I use Message Queue for Apache Kafka.
Causes
Possible causes are:
For clients earlier than version 0.10.2: The consumer does not have a separate thread to send heartbeat messages. Heartbeat messages are sent by using the poll interface. If the consumption response is stuck, the request to send heartbeat messages times out, and a rebalance occurs.
For clients of version 0.10.2 and later: No messages are pulled after the time specified by the
max.poll.interval.msparameter elapses. This causes the client to disconnect from the message queue and triggers a rebalance. The default value of the max.poll.interval.ms parameter is 5 minutes.
Solutions
You must understand the following parameters:
session.timeout.ms: specifies the timeout period of requests to send heartbeat messages. You can specify a custom value based on your business requirements.max.poll.records: specifies the maximum number of messages returned for each poll.For clients earlier than version 0.10.2: Heartbeat messages are sent by using the poll interface. The consumer does not have a separate thread to send heartbeat messages.
For clients of version 0.10.2 and later: You can set the
max.poll.interval.msparameter to prevent the clients from not consuming messages for a long time.
The following items provide references for you to set parameter values:
session.timeout.ms: For clients earlier than version 0.10.2, set a value larger than the time required to consume a batch of messages but smaller than 30 seconds. We recommend that you set the value to 25 seconds. For clients of version 0.10.2 and later, keep the default value of 10 seconds.
max.poll.records: Set a value far smaller than
the number of consumed messages per thread per second multiplied by the number of threads multiplied by the value of the max.poll.interval.ms parameter.max.poll.interval.ms: Set a value larger than
the value of max.poll.records divided by the product of the number of consumed messages per thread per second multiplied by the number of threads).
Improve the consumption speed of the client and allocate a separate thread to process the consumption logic.
Reduce the number of topics subscribed to by a group to no more than five. We recommend that one group subscribes to only one topic.
Upgrade the client to version 0.10.2 or later.