ApsaraMQ for RocketMQ:Migrate Apache RocketMQ clusters to ApsaraMQ for RocketMQ instances

Prerequisites

• An ApsaraMQ for RocketMQ 5.0 instance is created. For more information, see Create an instance.

• A service-linked role is created. For more information, see Service-linked roles.

  • Role name: AliyunServiceRoleForRMQMigration

  • Policy: AliyunServiceRolePolicyForRMQMigration

  • Description: Allow ApsaraMQ for RocketMQ to assume this role to access virtual private clouds (VPCs).

Usage notes

Before the migration, check whether scheduled messages and messages that fail to be retried exist in the Apache RocketMQ cluster. After the migration is complete, all consumer and producer client connections are switched to the ApsaraMQ for RocketMQ instance. This may cause the consumption failure of some scheduled messages or messages that fail to be retried. We recommend that you retain some consumer processes in the Apache RocketMQ cluster to consume scheduled messages and messages that fail to be retried.

Migration process

• Step 1: Evaluate the migration task

  Evaluate the risks of migration and the version and feature compatibility between the Apache RocketMQ cluster and the ApsaraMQ for RocketMQ instance to determine the scope of data that you want to migrate.

• Step 2: Configure the network

  Specify the information about the network and nodes of the Apache RocketMQ cluster. ApsaraMQ for RocketMQ connects to the network based on a minimum scope policy to grant permissions on traffic switching and traffic switch checking.

• Step 3: Migrate metadata

  ApsaraMQ for RocketMQ reads the metadata of topics and groups in the Apache RocketMQ cluster and copies the metadata to the destination ApsaraMQ for RocketMQ instance.

• Step 4: Change the endpoint

  Confirm all producers and consumers that are involved in the migration and change the endpoint in the messaging code of the source Apache RocketMQ cluster to the endpoint of the destination ApsaraMQ for RocketMQ instance.

• Step 5: Migrate messages

  Perform phased traffic switching by topic.

Step 1: Evaluate the migration task

Technical evaluation

• Limits on Apache RocketMQ clusters: Before you migrate an Apache RocketMQ cluster to an ApsaraMQ for RocketMQ instance, evaluate whether the cluster meets the requirements. If the cluster does not meet the requirements, submit a ticket. The following table describes the limits on Apache RocketMQ clusters.

  Limit	Description
  Deployed version	Apache RocketMQ 4.x and 5.x brokers are supported.
  Network	The source Apache RocketMQ cluster must be deployed in a VPC. If the source Apache RocketMQ cluster is deployed in an on-premises data center, make sure that the cluster can be accessed from a VPC.
  Region	You can migrate self-managed Apache RocketMQ clusters to ApsaraMQ for RocketMQ instances in the following regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Shenzhen), and China (Zhangjiakou).
  Feature	ApsaraMQ for RocketMQ does not support access control list (ACL) verification. If you enable the ACL feature for the Apache RocketMQ cluster, you must disable the feature before migration. If you do not disable the ACL feature, the feature does not take effect after migration.
  Parameter	Message size A message cannot exceed 4 MB in size. Message retention period Minimum value: 24. Unit: hours. Maximum value: 720. Unit: hours. Maximum delay time of scheduled messages Subscription and pay-as-you-go Standard Edition instances and serverless Standard Edition and Professional Edition instances: 7 days. Subscription and pay-as-you-go Professional Edition and Enterprise Platinum Edition instances: 40 days. For more information, see Quotas and limits.

• Limits on SDK versions: The migration solution follows the principle of minimal changes. In most cases, the versions of client SDKs are automatically upgraded. If the migration involves the upgrade of major versions, new features are added and stability optimizations are provided. In this case, we recommend that you upgrade the SDK version during migration.

  SDK	Programming language	SDK version	Upgrade required or not
  Apache RocketMQ Remoting SDK The following code shows the endpoint format of SDKs of this type: producer.setNamesrvAddr("xxx:9876"); consumer.setNamesrvAddr("xxx:9876");	Java	5.x SDK	This type of SDK is compatible with ApsaraMQ for RocketMQ instances. No upgrade is required.
  	Java and C++	4.x SDK	If the PullConsumer, DefaultLitePullConsumer, and DefaultPullConsumer operations are called by the source Apache RocketMQ cluster, you must upgrade the SDK to 5.x. For more information, see Overview.
  Apache RocketMQ gRPC SDK The following code shows the endpoint format of SDKs of this type: ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder() .setEndpoints(192.168.XX.XX:9876) .setCredentialProvider(sessionCredentialsProvider) .build();	Java, C++, Go, and C#	5.x SDK	This type of SDK is compatible with ApsaraMQ for RocketMQ instances. No upgrade is required.

Migration scope confirmation

The migration solution provided by ApsaraMQ for RocketMQ allows you to migrate Apache RocketMQ clusters to ApsaraMQ for RocketMQ instances by topic. Batch migration, canary release, and rollback are also supported. This effectively reduces risks caused by large-scale changes.

Step 2: Configure the network

In this step, you need to create a migration task and configure the network for the source Apache RocketMQ cluster. The migration tool provided by ApsaraMQ for RocketMQ can use the configured network to read metadata in the source Apache RocketMQ cluster and manage the migration task.

Usage notes

• The migration tool provided by ApsaraMQ for RocketMQ can access only the following information about the source Apache RocketMQ cluster based on the minimum scope policy:

  • The metadata of topics

  • The metadata of groups

  • Information about dynamic routing registration in topics

  • Information about consumer connections and message accumulation

  The migration tool does not access other information about the source Apache RocketMQ cluster or write configuration information to the cluster. Therefore, the access of the migration tool to an Apache RocketMQ cluster does not have impacts on the running of the cluster.

• Make sure that network configurations are valid before you go to the next step. Once you enter the next step, you cannot roll back to the network configuration step. To modify the network configurations, you must create a new migration task.

Procedure

1. Log on to the ApsaraMQ for RocketMQ console.

2. In the top navigation bar, select the region where the source Apache RocketMQ cluster and destination ApsaraMQ for RocketMQ instance reside. In the left-side navigation pane, choose RocketMQ Copilot > Migration to Cloud.

3. On the Migration to Cloud page, click Create Task.

4. In the Create Migration Task panel, configure the parameters and click OK.

   For information about the parameters, see Parameters configured for the network of the source Apache RocketMQ cluster.

5. In the Network Settings step of the Migration to Cloud wizard, enter the network information about the source Apache RocketMQ cluster and click Configure Network.

   For information the parameters, see Parameters configured for the network of the source Apache RocketMQ cluster.

6. Wait until the background configuration is complete. Then, click Next.

Parameter settings

Step 3: Migrate metadata

After you configure the network, you must select the topics and groups that you want to migrate based on your migration plan to complete the metadata migration.

Usage notes

• During metadata migration, the ApsaraMQ for RocketMQ migration tool dynamically reads the information about all topics and groups that are created in the Apache RocketMQ cluster and displays the information in lists. You need to migrate only the metadata of the topics and groups that you want to migrate.

• You cannot roll back to this step. Make sure that the topics and groups that you want to migrate are all migrated before you go to the next step. Otherwise, you must manually add topics in subsequent operations.

Procedure

1. In the Metadata Migration step, click the Topic Metadata tab.

2. Select the topic that you want to migrate from the topic list and the corresponding message type from the Message Type drop-down list. Then, click Confirm and Import in the Actions column to import the topic.

   You can also select multiple topics and click Batch Import to import the topics at a time.

   Important

   Message types are not defined in Apache RocketMQ 4.x. ApsaraMQ for RocketMQ verifies whether the message type that you specified for the topic to be migrated is consistent with the type of message that the topic handles. When you migrate metadata, you must select the message type based on your actual business.

   If you select a wrong message type, messages fail to be sent or received after migration. If you are not sure about the message type of the topic or whether different types of messages exist in the topic, submit a ticket.

3. Click the Group Metadata tab, select the group that you want to migrate from the group list, select the order in which messages are consumed from the Consumption Order drop-down list, and then click Confirm and Import in the Actions column to import the group.

   You can also select multiple groups and click Batch Import to import the groups at a time.

   Important

   The consumption order of messages in an Apache RocketMQ 4.x SDK is specified on the client. The consumption order of messages in a group on an ApsaraMQ for RocketMQ 5.x instance is controlled by the broker. When you migrate metadata, you must specify the consumption order of messages in the group based on your business scenario.

   If you select a wrong message consumption order, consumption exceptions occur after migration. If you are not sure about the consumption order of messages in a group, submit a ticket.

4. Click Next after you confirm that all topics and groups that you want to migrate are imported.

Step 4: Change the endpoint

After you migrate the metadata of the specified topics and groups, you start to migrate the online business. At this stage, you must change the endpoint that the producer and consumer applications use to access the source cluster to the endpoint that they use to access the destination ApsaraMQ for RocketMQ 5.x instance. This way, the producer and consumer applications that are involved in the migration task can be connected to the destination ApsaraMQ for RocketMQ instance.

Usage notes

• After you change the endpoint, you must restart the producer and consumer applications to connect the applications to the destination ApsaraMQ for RocketMQ instance. During this process, the migration tool routes topics that are used for messaging to the Apache RocketMQ cluster at the backend. The messaging system remains unchanged in this step. No limit is imposed on the order in which the messaging applications change the endpoint.

• Make sure that the endpoint is changed in all producer and consumer applications that are involved in the migration.

  • If the endpoint is not changed in specific producer applications, specific messages fail to be sent.

  • If the endpoint is not changed in specific consumer applications, message accumulation occurs and specific messages cannot be consumed.

Examples of endpoint change

• Apache RocketMQ Remoting SDKs

  • Before change:

    producer.setNamesrvAddr("192.168.XX.XX:9876");
      
    consumer.setNamesrvAddr("192.168.XX.XX:9876");

  • After change:

    producer.setNamesrvAddr("rmq-cn-pe334******-vpc.cn-hangzhou.rmq.aliyuncs.com:8080");
      
    consumer.setNamesrvAddr("rmq-cn-pe334******-vpc.cn-hangzhou.rmq.aliyuncs.com:8080");

• Apache RocketMQ gRPC SDKs

  • Before change:

    ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
                .setEndpoints(192.168.XX.XX:9876)
                .setCredentialProvider(sessionCredentialsProvider)
                .build();

  • After change:

    ClientConfiguration clientConfiguration = ClientConfiguration.newBuilder()
                .setEndpoints(rmq-cn-pe334******-vpc.cn-hangzhou.rmq.aliyuncs.com:8080)
                .setCredentialProvider(sessionCredentialsProvider)
                .build();

Procedure

Step 5: Migrate messages

To migrate messages, you must switch read and write traffic in each topic of the Apache RocketMQ cluster to the ApsaraMQ for RocketMQ instance.

Usage notes

• During traffic switching, you must check whether messages are sent and received as expected each time the topic status changes. If no exceptions occur, go to the next step. If exceptions occur, roll back to the previous step and troubleshoot the issue.

• Make sure that the traffic in all topics that you want to migrate are switched and messages can be sent and received as expected before you complete the migration task. You cannot modify the information about traffic switching after you complete the migration task.

Traffic switching details

Switch traffic

1. In the Message Migration step, select the topic for which you want to migrate messages and confirm the check status of the topic.

   • If the status is Check Passed, go to the next step.

   • If the status is not Check Passed, troubleshoot the exception based on the check result. Then, click Re-verify in the Actions column until the check is passed. After the check is passed, go to the next step.

     For information about items that are checked during each stage of traffic switching, see Check items.

   • If the status is not Check Passed and you confirm that the check result does not affect the migration, click Ignore Check in the Actions column of the topic and go to the next step.

2. In the Actions column of the topic whose traffic you want to switch, click Switch Traffic.

3. In the message that appears, click OK.

   Traffic switching consists of four stages. You must perform the traffic switching operation four times. Traffic switching is complete until Traffic Switching Stage changes to Read and Write in Destination Cluster.

   For information about the change of read and write traffic in topics during each stage of traffic switching, see Traffic switching status.

4. Confirm that the traffic of all topics that you want to migrate in the migration task is switched and click Migrated in the lower part of the page.

Other operations

The following items describe other operations that you can perform on the Message Migration page when you switch traffic:

• Roll Back

  • Roll back to the previous state: If the migration does not work as expected, you can roll back the status of the specified topic to a previous state in which the migration works as expected and perform subsequent operations after you troubleshoot the issue.

  • Roll back to the initial status: You can roll back to the initial status of traffic switching by using this method. The initial status of traffic switching refers to the routing state and is used for emergency troubleshooting.

    Note

    If you perform this operation, huge changes are caused to the status of traffic switching. Unconsumed messages that are produced during previous migration process may be delayed or cannot be processed.

• Create Topic

  If a topic that you want to migrate is not selected during metadata migration, you can create a topic that has the same name as the topic that you want to migrate on the ApsaraMQ for RocketMQ 5.x instance during traffic switching.

• Batch Traffic Switching/Batch Rollback

  You can perform batch traffic switching and batch rollback.

  Note

  You can perform batch traffic switching and batch rollback only for topics that are at the same traffic switching stage.

Check items during traffic switching

References

• For information about the differences between open source Apache RocketMQ and ApsaraMQ for RocketMQ and the migration mechanism and benefits of cloud migration, see Overview.

• After the migration task is complete, you can use metrics on the ApsaraMQ for RocketMQ dashboard to check whether the instance is running as expected and whether the business data of the instance is normal. If exceptions occur, you can roll back the task. For more information, see Dashboard and Roll Back.