If you use a self-managed Apache Kafka cluster and want to reduce costs, increase scalability, and improve reliability, you can migrate the cluster to an ApsaraMQ for Kafka instance. ApsaraMQ for Kafka is a fully managed and O&M-free service that allows you to focus on business development.
Usage notes
After the migration is complete, the source topics and groups in the self-managed Apache Kafka cluster are not deleted. Instead, new topics and groups with the same configurations are created in the destination ApsaraMQ for Kafka instance.
During the migration, only the configurations of topics and groups are migrated.
When you migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance, you can use the cloud migration feature in the ApsaraMQ for Kafka console or a migration tool. We recommend that you use the cloud migration feature to migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance and use a migration tool only when the public endpoint is unavailable for a self-managed Kafka cluster.
Solution description
When you migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance, you must use the original consumers to consume all messages that are queued in the original cluster. Producers and consumers are deployed in clusters. You can migrate servers that are deployed in the clusters one by one. The migration process does not affect your upper-layer services.
The following figure shows how to migrate a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance.
ApsaraMQ for Kafka provides the following migration methods:
Method 1: Use the cloud migration feature
Obtain the public endpoint of the self-managed Apache Kafka cluster.
Create a cloud migration task in the ApsaraMQ for Kafka console.
Create a consumer group on the ApsaraMQ for Kafka instance to consume messages that are sent to the instance.
Create a producer on the ApsaraMQ for Kafka instance, terminate the producer in the self-managed Apache Kafka cluster, and then use the original consumer group to consume messages that are queued in the self-managed Apache Kafka cluster.
After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer group, terminate the consumer group and the self-managed Apache Kafka cluster.
Method 2: Use a migration tool
Use a migration tool to migrate data in the self-managed Apache Kafka cluster to the ApsaraMQ for Kafka instance.
Create a consumer group on the ApsaraMQ for Kafka instance to consume messages that are sent to the instance.
Create a producer on the ApsaraMQ for Kafka instance, terminate the producer in the self-managed Apache Kafka cluster, and then use the original consumer group to consume messages that are queued in the self-managed Apache Kafka cluster.
After all messages in the self-managed Apache Kafka cluster are consumed by the original consumer group, terminate the consumer group and the self-managed Apache Kafka cluster.
Use the cloud migration feature
ApsaraMQ for Kafka provides a fully managed and O&M-free cloud migration feature that allows you to synchronize data from a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance or synchronize data from an ApsaraMQ for Kafka instance to another ApsaraMQ for Kafka instance in the ApsaraMQ for Kafka console without the need to use a tool. After the migration, the metadata of the destination cluster is the same as the metadata of the source cluster and can be constantly updated. For more information, see Migrate a self-managed or cross-cloud instance to an ApsaraMQ for Kafka instance.
Use a migration tool
Migration tools
Migration tool | Description | References |
|
| |
You can use this tool to copy the data image of a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance. |
Migration process
The following points describe the migration process:
(Optional) Evaluate instance specifications: If you know the information about the self-managed Apache Kafka cluster that you want to migrate, such as the traffic, disk capacity, disk type, and partition number, you can evaluate the specifications of the ApsaraMQ for Kafka instance that you want to purchase.
If you do not know the information about the self-managed Apache Kafka cluster that you want to migrate, you can skip this step. If you migrate the metadata of the self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance, you can specify a migration tool to collect the information about the cluster. ApsaraMQ for Kafka automatically evaluates the specification requirements of the destination instance based on the information that is collected by the migration tool.
Migrate the self-managed Apache Kafka cluster: You can run a command in the command-line interface (CLI) to migrate a specific type of resource in the self-managed Apache Kafka cluster or create a migration task to migrate all resources in the cluster.
Migrate the metadata of a self-managed Kafka cluster to a new ApsaraMQ for Kafka instance: In the ApsaraMQ for Kafka console, create a migration task, purchase an ApsaraMQ for Kafka instance based on the recommendation, and then deploy the instance and start the migration task.
NoteIf the metadata file contains traffic, disk, and configuration information about the self-managed Apache Kafka cluster, ApsaraMQ for Kafka automatically evaluates and recommends the specifications based on the metadata file. You can purchase an instance based on the recommendation.
Migrate the metadata of a self-managed Apache Kafka cluster to an existing ApsaraMQ for Kafka instance: In the ApsaraMQ for Kafka console, create a migration task to migrate the metadata of the self-managed Apache Kafka cluster to an existing ApsaraMQ for Kafka instance.
Migrate a specific type of resource: If you want to migrate only topics, groups, or messages in the self-managed Apache Kafka cluster, you can use the corresponding migration method.
Migrate topics from a self-managed Kafka cluster to an ApsaraMQ for Kafka instance
Migrate groups from a self-managed Kafka cluster to an ApsaraMQ for Kafka instance
(Optional) Migrate messages from a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance
ImportantAfter messages in message queues are consumed, the messages are not reused. We recommend that you do not migrate messages from a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance unless you need to back up the messages.
View the migration progress: ApsaraMQ for Kafka allows you to view the migration progress.
Verify the migration result: After the migration is successful, you can view the resource list.
References
For information about the benefits of migrating a self-managed Apache Kafka cluster to an ApsaraMQ for Kafka instance, see Benefits.