All Products
Search

This Product

    ApsaraMQ for Kafka:Rebalance topic traffic

    Document Center

    ApsaraMQ for Kafka:Rebalance topic traffic

    Last Updated:Dec 11, 2024

    When you upgrade the traffic specification of an ApsaraMQ for Kafka instance, the corresponding cluster may be scaled out. After the cluster is scaled out, you must rebalance the topic traffic to evenly distribute the traffic across the brokers of the scaled-out cluster. Otherwise, the original topic traffic is still distributed across the brokers that are in the cluster before the scale-out. The original topics are subject to the maximum traffic purchased before the scale-out. New topics are not subject to the maximum traffic purchased before the scale-out.

    Prerequisites

    Your ApsaraMQ for Kafka instance is in the Running (Pending Rebalancing) state.

    Note

    For information about how to upgrade the traffic specification of an ApsaraMQ for Kafka instance and the rules for triggering a cluster scale-out, see Upgrade instance configurations.

    Usage notes

    If an ApsaraMQ for Kafka instance is in the Running (Pending Rebalancing) state, you can use this instance to send and receive messages. However, you cannot create resources, such as topics and groups, on this instance. To create resources, you must wait until the topic traffic is rebalanced. You can also select not to rebalance the topic traffic.

    Traffic rebalancing methods

    Traffic rebalancing method

    Working mechanism

    Impact

    Scenario

    Duration

    Add Partitions to All Topics

    After the cluster is scaled out, the system adds partitions to the new brokers for all topics on the original brokers.

    • New messages in partitions are out of order.

    • The number of partitions changes. If your client cannot automatically detect new partitions, you may need to restart the client or modify the client code. This may occur in scenarios in which extract, transfer, and load (ETL) is performed, or in scenarios in which you send messages to or consume messages from specified partitions.

    • The order of partitions is not required.

    • The partition to which messages are sent is not specified.

    • The selected consumption mode is subscription.

    Within seconds.

    Migrate Data from Partitions of All Topics (Recommended)

    • Local storage: The kafka-reassign-partitions tool is used to migrate topic data among partitions.

    • Cloud storage: The mapping is modified, but the topic data is not migrated among partitions.

    Note

    You can specify local storage as the storage engine only when you create topics for Professional Edition instances.

    • Local storage: Temporary internal traffic is generated.

    • Cloud storage: No temporary internal traffic is generated.

    All scenarios in which a cluster is scaled out.

    • Local storage: within minutes or hours. The actual duration depends on the amount of data that you want to migrate from the local storage. If the data volume is large, the migration may require several hours or longer to complete. We recommend that you migrate data during off-peak hours.

    • Cloud storage: within seconds. A topic requires approximately 30 seconds to be migrated.

    Do Not Rebalance (Not Recommended)

    You do not need to perform operations. The original topics are still distributed on the brokers of the cluster before the scale-out, and new topics are evenly distributed across all brokers after the scale-out.

    • The original topics are subject to the traffic specification purchased before the scale-out.

    • If the volume of the original topic traffic is large, the traffic may be unbalanced among brokers.

    • The volume of the original topic traffic is small and not greatly increased after the cluster is scaled out.

    • Topics are created after the cluster is scaled out. Most of the traffic is directed to the new topics.

    The configuration immediately takes effect.

    Procedure

    Note

    Topic rebalancing is automatically triggered on serverless ApsaraMQ for Kafka instances. You do not need to perform operations in the ApsaraMQ for Kafka console.

    1. Log on to the ApsaraMQ for Kafka console. In the Resource Distribution section of the Overview page, select the region where the ApsaraMQ for Kafka instance that you want to manage resides.

    2. On the Instances page, click the name of the instance that you want to manage.

    3. On the Instance Details page, click Rebalance Topic Traffic in the upper-right corner of the Overview section.

    4. In the Rebalance Topic Traffic for Instance panel, configure the Traffic Rebalancing Method parameter. For information about the supported traffic rebalancing methods, see Traffic rebalancing method.

      After you select a traffic rebalancing method, all topics on the instance enter the Pending Rebalancing state. You can check the topic status in the Status column on the Topics page.

    Execution result

    After the topic traffic is rebalanced, the topics enter the Running state. You can check the topic status in the Status column on the Topics page.