Client parameters of Message Queue for Apache Kafka - ApsaraMQ for Kafka

This topic describes the rules for configuring client parameters of Message Queue for Apache Kafka. We recommend that you specify proper values for these parameters based on your business requirements.

The following tables describe parameters for different clients:

Table 1. Producer client parameters
Parameter	Description
retries	The number of retries that is allowed for a message that fails to be sent.
retry.backoff.ms	The interval between retries when a message fails to be sent. We recommend that you set this parameter to 1000. Unit: milliseconds.
acks	The persistence mechanism for sending messages. To improve message sending performance, we recommend that you set this parameter to `1`. If you set this parameter to `0`, no response is returned from the broker. In this mode, the message sending performance is high, but the risk of data loss is also high. If you set this parameter to `1`, a response is returned when data is written to the leader. In this mode, the message sending performance and the risk of data loss are moderate. If a failure occurs on the leader, data loss may occur. If you set this parameter to `all`, a response is returned when data is written to the leader and synchronized to the followers. In this mode, the message sending performance is low, but the risk of data loss is low. Data loss occur only when the leader and the followers fail at the same time.
batch.size	The volume of cached messages to be delivered to each partition. When the volume of cached messages reaches the specified upper limit, a network request is triggered. Then, the producer sends the messages to the broker in a batch. If you set the batch.size parameter to a small value, the message sending performance and stability may be affected. We recommend that you use the default value 16384. Unit: bytes.
linger.ms	The maximum storage duration for each message in the cache. If a message is stored in the cache for a period of time that exceeds the maximum storage duration, the producer immediately sends the message to the broker without using the batch.size parameter. We recommend that you set the linger.ms parameter to a value that ranges from 100 to 1000 based on your business requirements. Unit: milliseconds.
partitioner.class	The strategy that you use for partitioning. We recommend that you use the sticky partitioning strategy to improve message sending performance. If the version of your producer client is 2.4 or later, the sticky partitioning strategy is used by default.
buffer.memory	The size of memory pool on the client. If you set this parameter to a small value, the time that is consumed to apply for memory may be long. This causes low message sending performance or message sending timeout. We recommend that you set this parameter to a value that is equal to or greater than the value of `Value of the batch.size parameter × Number of partitions × 2`. Unit: bytes.

Table 2. Consumer client parameters
Parameter	Description
fetch.min.bytes	The minimum amount of data that the consumer can pull from the broker. Before you configure this parameter, you must evaluate the number of messages to be sent by the producer. If you set this parameter to an excessively large value, message delay may be increased when the consumer consumes messages. If you set this parameter to an excessively small value, the frequency at which the consumer pulls messages may be increased. Unit: bytes.
fetch.max.wait.ms	The maximum amount of time that the broker waits to return a response. Unit: milliseconds. If you use local storage and configure the fetch.min.bytes parameter, the broker returns a response after the value that is specified by the fetch.min.bytes parameter is reached. After the time that is specified by the fetch.max.wait.ms parameter elapses, a response is returned regardless of whether the value that is specified by the fetch.min.bytes parameter is reached. If you use cloud storage, the broker immediately returns a response once new data is sent in instead of waiting until the time that is specified by the fetch.min.bytes parameter elapses.
max.partion.fetch.bytes	The maximum amount of data that each partition returns. Unit: bytes.
session.timeout.ms	The interval at which the consumer sends heartbeats. If no heartbeat is sent within the interval, the broker determines that the consumer is dead, and a rebalance is triggered. During the rebalance, the client stops consuming data and waits for the rebalance to complete. We recommend that you set this parameter to a value that ranges from 30000 to 60000. Unit: milliseconds. Valid values: 6000 to 300000.
max.poll.records	The maximum number of messages returned in a single call to poll(). If you set this parameter to an excessively large value, you must process the message consumption logic at the earliest opportunity. If the message consumption logic is processed at a low speed, data in the next call to poll() is affected. As a result, no heartbeat is sent during the interval that is specified by the session.timeout.ms parameter, and a rebalance is triggered. We recommend that you set this parameter to a value that is smaller than the value of `<Number of messages consumed by each thread per second> × <Number of threads consuming messages> × <Value specified by the session.timeout.ms parameter, in second>`. Important In Java Client 0.10.1 and later versions, a separate thread is used to send heartbeats. If you use a Java Client of earlier versions or a client in another language, you must specify proper values for parameters that are related to data processing time and heartbeat sending interval. This helps prevent frequent rebalances from affecting normal consumption.
max.poll.interval.ms	The maximum interval between invocations of poll(). This parameter is required only in Java client 0.10.1 and later versions. If the consumer does not call poll() during the interval that is specified by the session.timeout.ms parameter, the broker determines that the consumer is dead, and a rebalance is triggered. Therefore, you must set this parameter to a proper value. We recommend that you set this parameter to a value that is greater than the value of `<Time consumed to consume a message> × <Number of data records pulled by poll()>`. In most cases, you can use the default value. Unit: milliseconds. Default value: 300000.
enable.auto.commit	Specifies whether to automatically commit the consumer offset. Valid values: true false Default value: true.
auto.commit.interval.ms	The interval at which the consumer offsets are automatically committed. Default value: 1000. Unit: milliseconds.
auto.offset.reset	The policy that is used to reset the consumer offset. latest: resets the consumer offset to the latest offset. earliest: resets the consumer offset to the earliest offset. none: does not reset the consumer offset. Note We recommend that you set this parameter to latest instead of earliest. This way, the consumer does not consume messages from the start when an invalid offset exists. This helps prevent the consumer from repeatedly consuming messages. If you manually manage the consumer offset, you can set this parameter to none.