Limits of ApsaraMQ for Kafka - ApsaraMQ for Kafka - Alibaba Cloud Documentation Center

This topic describes the limits that are imposed on the metrics of ApsaraMQ for Kafka. To prevent program exceptions, make sure that the setting of a specific configuration item is within the corresponding limit when you use ApsaraMQ for Kafka.

Important

The service level agreement (SLA) does not apply to instability that occurs because your instance configurations exceed the limits.

Limit description

The following table describes the limits of ApsaraMQ for Kafka.

Item	Limit	Description
Maximum number of topics or partitions	Supported	Messages in ApsaraMQ for Kafka are stored and coordinated by partition. If an excessive number of topics or partitions exist, storage fragmentation occurs. This reduces the performance and stability of your cluster.
Decrease in the number of partitions in a topic	Not supported	This is due to the constraints in the design of Apache Kafka.
Exposed ZooKeeper	Not supported	You do not need to access ZooKeeper on your client to use Apache Kafka V0.9.0 and later. In ApsaraMQ for Kafka, ZooKeeper is partially shared. For security purposes, ZooKeeper is not exposed. You do not need to understand how ZooKeeper works.
Logon to the machine on which ApsaraMQ for Kafka is deployed	Not supported	None.
Supported versions	2.2.x to 3.3.x	Non-serverless instances: 2.2.x to 2.6.x. Serverless instances: 3.3.x. For information about how to upgrade instance versions, see Upgrade instance versions.
Partition-to-topic ratio	1:1	The number of available topics on an ApsaraMQ for Kafka instance is equal to the total number of partitions on the instance. For example, you select the alikafka.hw.2xlarge specification and purchase 50 partitions for your instance. The total number of partitions on the instance is 1,050 (50 purchased and 1,000 provided free of charge). The number of available topics on the instance is 1,050. Note This limit applies only to non-serverless instances.
Change in the region of an instance	Not supported	After an instance is purchased and deployed, the region of the instance is closely associated with the physical resources and cannot be changed. If you want to change the region of an instance, release the instance and purchase a new instance.
Change in the network properties of an instance	Supported	You can modify the network properties of an instance based on your business requirements. For more information, see Upgrade instance configurations.
Message size	10 MB	The maximum size of a message is 10 MB. Messages larger than 10 MB cannot be sent.
Monitoring and alerting	Supported	The data has a latency of 1 minute.
Endpoint	Determined by the purchased edition	Standard Edition instances: support the default and Secure Sockets Layer (SSL) endpoints. Professional Edition instances: support the default, SSL, and Simple Authentication and Security Layer (SASL) endpoints. Professional Edition instances: support the default, SSL, and Simple Authentication and Security Layer (SASL) endpoints.
Single-partition topics that use cloud storage	Supported. However, single-partition topics that use cloud storage may be unavailable during breakdowns and upgrades.	We recommend that you create topics that have more than one partition. If your business requires single-partition topics, we recommend that you use local storage. Note This limit applies only to non-serverless instances. Single-partition topics that use cloud storage on serverless instances feature high availability. You can set the Storage Engine parameter to Local Storage only when you create a topic on a Professional Edition instance.

Note

You can no longer purchase non-serverless ApsaraMQ for Kafka instances by topic. If your existing instances were purchased by topic, the number of available partitions is the number of topics multiplied by 16. For a Professional Edition instance purchased by topic, the number of available topics is twice the number of purchased topics.

Quota limits

The following table describes the quota limits of ApsaraMQ for Kafka. If the limits are exceeded, stability issues may occur. "Other limits" in the table refer to scenarios in which system instability may be caused by broker overload. Exercise caution in these scenarios.

The following limits are applicable to clusters unless otherwise stated. If you want to increase the quotas, submit a ticket.

Item	Limit	Description
Number of connections on each node	The minimum number of connections on an instance is 1,000. In this case, the traffic specification provides a maximum traffic of 20 MB/s. The number of connections on an instance increases with the traffic specification. Each time the amount of traffic in the traffic specification increases by 100 MB/s, the number of connections increases by 1,000. The maximum number of connections on an instance is 10,000. You can use the following formula to calculate the number of connections on an instance: `C = min (10,000, 1,000 + (F//100) × 1,000)`. In the preceding formula, F indicates the amount of traffic in MB/s, C indicates the number of connections, and // indicates that the number is rounded down to the largest integer that is smaller than or equal to the number.	The number of TCP connections on a broker. If you want to increase the quota, submit a ticket. Reserved capacity for message publishing on a serverless instance = Amount of traffic in the traffic specification × 3.
Number of connections on each node over the Internet	The minimum number of connections on an instance is 200. In this case, the traffic specification provides a maximum traffic of 20 MB/s. The number of connections on an instance increases with the traffic specification. Each time the amount of traffic in the traffic specification increases by 100 MB/s, the number of connections increases by 100. The maximum number of connections on an instance is 1,000. You can use the following formula to calculate the number of connections on an instance: `C = min (1,000, 200 + (F//100) × 100)`. In the preceding formula, F indicates the amount of traffic in MB/s, C indicates the number of connections, and // indicates that the number is rounded down to the largest integer that is smaller than or equal to the number.	The number of TCP connections on a broker over the Internet. If you want to increase the quota, submit a ticket.
Connection frequency on each node	Serverless instance: 150 connections per second. Non-serverless instance: 50 connections per second.	The number of attempts to establish a connection between the client and the broker per second, including failed connections caused by authentication failures.
Connection frequency on each node over the Internet	10 connections per second.	The number of attempts to establish a connection between the client and the broker over the Internet per second, including failed connections caused by authentication failures.
Batch size	If the 50th percentile of the batch size of messages is 4 KB, fragmented transmission occurs.	The batch size of messages included in a produce request after messages are sent in a batch from the client. We recommend that you use clients whose version is later than 2.4 to improve batch sending performance. For more information, see Send messages in batches to improve message sending performance.
Frequency of produce requests in a cluster	The minimum number of produce requests per second in a cluster is 10,000. In this case, the traffic specification provides a maximum traffic of 20 MB/s. The number of produce requests per second in a cluster increases with the traffic specification. Each time the amount of traffic in the traffic specification increases by 20 MB/s, the number of produce requests per second increases by 5,000. You can use the following formula to calculate the number of produce requests per second in a cluster: `R = 10,000 + (F//20) × 5,000)`. In the preceding formula, F indicates the amount of traffic in MB/s, R indicates the number of produce requests per second, and // indicates that the number is rounded down to the largest integer that is smaller than or equal to the number.	The number of produce requests that are sent by clients in a cluster. If you want to increase the quota, submit a ticket.
Frequency of fetch requests in a cluster	The minimum number of fetch requests per second in a cluster is 5,000. In this case, the traffic specification provides a maximum traffic of 20 MB/s. The number of fetch requests per second in a cluster increases with the traffic specification. Each time the amount of traffic in the traffic specification increases by 20 MB/s, the number of fetch requests per second increases by 2,500. You can use the following formula to calculate the number of fetch requests per second in a cluster: `R = 5,000 + (F//20) × 2,500)`. In the preceding formula, F indicates the amount of traffic in MB/s, R indicates the number of fetch requests per second, and // indicates that the number is rounded down to the largest integer that is smaller than or equal to the number.	The number of fetch requests that are received by clients. If you want to increase the quota, submit a ticket.
Frequency of offset commit on a node	The minimum number of offset commit requests per second on a node is 100. In this case, the traffic specification provides a maximum traffic of 20 MB/s. The number of offset commit requests per second on a node increases with the traffic specification. Each time the amount of traffic in the traffic specification increases by 20 MB/s, the number of offset commit requests per second increases by 100. The maximum number of offset commit requests per second on a node is 1,000. You can use the following formula to calculate the number of offset commit requests on a node: `R = min (1,000, 100 + (F//100) × 100)`. In the preceding formula, F indicates the amount of traffic in MB/s, R indicates the number of offset commit requests per second, and // indicates that the number is rounded down to the largest integer that is smaller than or equal to the number.	The number of offset commit requests that are sent by clients. If you want to increase the quota, submit a ticket.
Frequency of metadata requests in a cluster	The minimum number of metadata requests per second in a cluster is 100. In this case, the traffic specification provides a maximum traffic of 20 MB/s. The number of metadata requests per second in a cluster increases with the traffic specification. Each time the amount of traffic in the traffic specification increases by 100 MB/s, the number of metadata requests per second increases by 100. The maximum number of metadata requests per second in a cluster is 1,000. You can use the following formula to calculate the number of metadata requests in a cluster: `R = min (1,000, 100 + (F//100) × 100)`. In the preceding formula, F indicates the amount of traffic in MB/s, R indicates the number of metadata requests per second, and // indicates that the number is rounded down to the largest integer that is smaller than or equal to the number.	The number of metadata requests that are received by clients. Examples: `METADATA, INIT_PRODUCER_ID, and CREATE_ACL, JOIN_GROUP`. Warning If a large number of metadata requests are initiated in a second, cluster instability may occur.
Maximum number of partitions in a cluster	Serverless instance: For an instance whose reserved capacity for message publishing is less than or equal to 1 GB/s, 3,000 partitions are provided. For an instance whose reserved capacity for message publishing is greater than 1 GB/s, the number of partitions increases by 300 each time the reserved capacity for message publishing increases by 100 MB/s. The maximum number of partitions on an instance is 15,000. For information about the number of partitions on a non-serverless ApsaraMQ for Kafka instance, see Subscription.	The number of partitions includes all partitions in different types of topics on an instance. If you want to increase the quota, submit a ticket.
Frequency of partition creation and deletion	You can create or delete up to 900 partitions every 10 seconds.	The partitions include all partitions that are created or deleted in the ApsaraMQ for Kafka console and by calling API operations and using Kafka Admin.
Number of consumer groups in a cluster	You can create up to 2,000 consumer groups in a cluster. The ratio of the number of subscribed topics to the number of consumer groups cannot exceed 3:1. We recommend that you set the ratio to 1:1.	The number of consumer groups that you use. If you want to increase the quota, submit a ticket. Warning An excessive number of consumer groups may increase the workload on the broker and reduce metadata management efficiency. This affects performance and increases the duration of failure recovery.
Message format version	The message format must be of version 1 or later.	We recommend that you use clients whose version is later than 2.4. Warning If you use an earlier version of the message format, issues such as increase in broker CPU, decrease in throughput performance, and incompatibility may occur.
Other limits	If you enable specific compression algorithms, the broker performance and throughput may be affected. If you initialize `Producer Id` of a large number of high-frequency transactions, out of memory (OOM) exceptions occur. This causes overload on the broker and affects the system stability. Therefore, the value of the `transactional.id.expiration.ms` parameter is changed to 15 minutes. If you want to increase the value, submit a ticket. Invalid timestamps in message bodies are intercepted. If you set the `message.timestamp.type` parameter to `CreateTime`, a message is rejected if the difference between the timestamp when the broker receives the message and the timestamp specified in the message exceeds the threshold specified by the `message.timestamp.difference.max.ms` parameter. This setting is used to prevent invalid timestamp configuration. If the value of a timestamp is excessively small, LogSegment is immediately deleted. If the value of a timestamp is excessively large, LogSegment cannot be deleted. Write errors in compacted topics can cause insufficient storage space in the cluster. To prevent failovers caused by write errors in compacted topics, the maximum storage space in the partitions of a compacted topic is 5 GB. If you require a larger storage space, submit a ticket.	None.