This issue may occur due to one of the following causes: The consumption traffic reaches the network bandwidth, the traffic for a single message exceeds the network bandwidth, and the traffic for messages pulled by the consumer at the same time exceeds the network bandwidth.
Issue description
Topics to which a consumer subscribes receive messages, and the consumption does not reach the latest offset. However, the consumer slowly pulls messages from the broker or fails to pull messages from the broker. This issue occurs more frequently when the consumption is performed over the Internet.
Possible causes
This issue may occur due to the following causes:
The consumption traffic reaches the network bandwidth.
The traffic for a single message exceeds the network bandwidth.
The traffic for messages pulled by the consumer at the same time exceeds the network bandwidth.
NoteThe following parameters determine the number and size of messages that a consumer can pull at the same time:
max.poll.records: the maximum number of messages that the consumer can pull at the same time.
fetch.max.bytes: the maximum number of bytes of messages that the consumer can pull at the same time.
max.partition.fetch.bytes: the maximum number of bytes of messages that the consumer can pull from a single partition at the same time.
Solutions
Log on to the ApsaraMQ for Kafka console to query messages.
If messages are returned in the query result, perform the subsequent steps.
In the left-side navigation pane of the Instances page, choose . On the page that appears, click the Monitoring Chart tab and view the instance_internet_rx.rate(bit/s) chart to check whether the consumption traffic of the instance reaches the network bandwidth.
If the consumption traffic reaches the network bandwidth, increase the network bandwidth.
Check whether the traffic for a single message in the topic exceeds the network bandwidth.
If the traffic exceeds the network bandwidth, increase the network bandwidth or reduce the size of the message.
Check whether the traffic for messages pulled by the consumer at the same time exceeds the network bandwidth.
If the traffic exceeds the network bandwidth, change the configurations of the following parameters:
fetch.max.bytes: Set the parameter to a value smaller than the network bandwidth.
max.partition.fetch.bytes: Set the parameter to a value smaller than the limit value. The limit value is calculated by using the following formula: Limit value = Network bandwidth/Number of partitions that subscribe to messages.
If the consumer accesses the broker in a virtual private cloud (VPC), network bandwidth in this topic refers to the maximum write traffic of elastic network interfaces (ENIs) of the instance.
If the consumer accesses the broker over the Internet, network bandwidth in this topic refers to the Internet bandwidth of the instance.