Migrate a group from a self-managed Kafka cluster to ApsaraMQ for Kafka - ApsaraMQ for Kafka

This topic describes how to use the migration tool provided by ApsaraMQ for Kafka to migrate a group from a self-managed Kafka cluster to a ApsaraMQ for Kafka instance.Group

Prerequisites

Make sure that the following operations are complete:

Java Development Kit (JDK) 8 is downloaded and installed. For more information, see Java Downloads.
The migration tool is downloaded and installed. If the migration tool is not downloaded, click kafka-migration-assessment.jar to download the migration tool.
A ApsaraMQ for Kafka instance is purchased and deployed.
- For information about how to purchase an instance and deploy the instance in a virtual private cloud (VPC), see Purchase and deploy an instance that allows access from a VPC.
- For information about how to purchase an instance that can be accessed over the Internet and a VPC and deploy the instance, see Purchase and deploy an instance that allows access from the Internet and a VPC.

Procedure

Open your CLI.
Run the cd command to go to the directory in which the migration tool is stored.

Create a configuration file named kafka.properties.

The system defines subscription relationships and obtains the consumer offset of the original consumer group based on the settings that you configured in the kafka.properties file. The configuration file contains the following information:

## The endpoint. 
bootstrap.servers=localhost:9092

## The ID of the group. Group Make sure that the group does not contain consumer offset information. If the group contains consumer offset information, the group may not consume messages from the first message that needs to be consumed.Group 
group.id=XXX

## If you do not require certificate authorization, you do not need to configure the following parameters. 

## The Simple Authentication and Security Layer (SASL) mechanism. 
#sasl.mechanism=PLAIN

## The access protocol. 
#security.protocol=SASL_SSL

## The path of the root Secure Sockets Layer (SSL) certificate. 
#ssl.truststore.location=/Users/***/Documents/code/aliware-kafka-demos/main/resources/kafka.client.truststore.jks

## The password of the root SSL certificate. 
#ssl.truststore.password=***

## The path of the configuration file that you want to use for SASL. 
#java.security.auth.login.config=/Users/***/kafka-java-demo/vpc-ssl/src/main/resources/kafka_client_jaas.conf

Run the following command to check the information about the group that you want to migrate:Group

java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromTopic  \ 
--propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties  \ 
--destAk <yourAccessKeyId>  \ 
--destSk <yourAccessKeySecret>  \ 
--destRegionId <yourRegionId>  \ 
--destInstanceId <yourInstanceId>

Parameter	Description
propertiesPath	The path of the kafka.properties configuration file.
destAk	The AccessKey ID of the Alibaba Cloud account to which the destination ApsaraMQ for Kafka instance belongs.
destSk	The AccessKey secret of the Alibaba Cloud account to which the destination ApsaraMQ for Kafka instance belongs.
destRegionId	The ID of the region where the destination ApsaraMQ for Kafka instance is deployed.
destInstanceId	The ID of the destination ApsaraMQ for Kafka instance.

Check the information that is returned. The returned information describes the group that is migrated.

15:29:45 INFO - Will create consumer groups:[XXX, test-consumer-group]

Run the following command to commit the task to migrate the topic: Group

java -jar kafka-migration-assessment.jar ConsumerGroupMigrationFromTopic  \ 
--propertiesPath /usr/local/kafka_2.12-2.4.0/config/kafka.properties  \ 
--destAk <yourAccessKeyId>  \ 
--destSk <yourAccessKeySecret>  \ 
--destRegionId <yourRegionId>  \ 
--destInstanceId <yourInstanceId>  \
--commit

Parameter	Description
commit	Commits the migration task.

After you commit the migration task, the following information is returned:

15:35:51 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"C9797848-FD4C-411F-966D-0D4AB5D12F55","success":true,"message":"operation success"}
15:35:51 INFO - ConsumerCreate success, consumer group=XXX
15:35:57 INFO - cmd=ConsumerGroupMigrationFromTopic, request=null, response={"code":200,"requestId":"3BCFDBF2-3CD9-4D48-92C3-385C8DBB9709","success":true,"message":"operation success"}
15:35:57 INFO - ConsumerCreate success, consumer group=test-consumer-group

After you commit the task to migrate the topic to the ApsaraMQ for Kafka instance, you can log on to the ApsaraMQ for Kafka console and view information about the migration task on the Migration to Cloud page.