ApsaraDB for ClickHouse:Upgrade the major engine version by migrating data

Prerequisites

• The source cluster and the destination cluster are ApsaraDB for ClickHouse Community-compatible Edition clusters.

• The usernames and passwords of database accounts are created for the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster.

• The replicas of the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster are consistent.

• The source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster are deployed in the same region and use the same virtual private cloud (VPC).

• If you enable tiered storage of hot data and cold data for the source cluster, you need to enable tiered storage of hot data and cold data for the destination cluster. If you disable tiered storage of hot data and cold data for the source cluster, you need to disable tiered storage of hot data and cold data for the destination cluster.

• Each local table in the source ApsaraDB for ClickHouse cluster corresponds to a unique distributed table.

• The available storage capacity of the disk for the destination ApsaraDB for ClickHouse cluster is greater than or equal to 1.2 times the used storage capacity of the disk for the source ApsaraDB for ClickHouse cluster.

• The IP address of the source ApsaraDB for ClickHouse cluster is added to the IP address whitelist of the destination ApsaraDB for ClickHouse cluster.

• The IP address of the destination ApsaraDB for ClickHouse cluster is added to the IP address whitelist of the source ApsaraDB for ClickHouse cluster.

Usage notes

• You cannot downgrade the major engine version after the major engine version is upgraded.

• Before you migrate data, make sure that no management operations, such as scale-out, upgrade, or downgrade operations, are being performed on the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster.

• Full data migration and incremental data migration are supported between ApsaraDB for ClickHouse clusters.

• For non-MergeTree tables such as external tables and log tables, only the schemas are migrated. The data is not migrated. Therefore, if the source ApsaraDB for ClickHouse cluster contains non-MergeTree tables and you migrate data from the source ApsaraDB for ClickHouse cluster to the destination ApsaraDB for ClickHouse cluster, the non-MergeTree tables in the destination ApsaraDB for ClickHouse cluster have only a table schema and no specific business data.

• After the major engine version is upgraded, the client fails to connect to the database service. You must change the endpoint for connecting the client to the database service.

Impacts of the major engine version upgrade

• If you upgrade the major engine version for an ApsaraDB for ClickHouse cluster of version 20.8 or later by migrating data, you can read data from or write data to the source ApsaraDB for ClickHouse cluster during the data migration. To make sure that the migration task is complete as soon as possible, we recommend that you stop writing data to the source ApsaraDB for ClickHouse cluster when about 95% of the data has been migrated. If you upgrade the major engine version for an ApsaraDB for ClickHouse cluster of version 20.3 or earlier by migrating data, you can only read data from the source ApsaraDB for ClickHouse cluster during the data migration. After the data migration is complete, you can read data from or write data to the source ApsaraDB for ClickHouse cluster.

• You cannot read data from or write data to the destination ApsaraDB for ClickHouse cluster until the data migration is complete.

Supported versions

The following table describes the supported upgrades for each major engine version of the source ApsaraDB for ClickHouse cluster.

Step 1: Create a destination ApsaraDB for ClickHouse cluster

Create an ApsaraDB for ClickHouse Community-compatible Edition cluster of a later version. For more information about the supported versions and the procedure for creating an ApsaraDB for ClickHouse Community-compatible Edition cluster, see the Supported versions section of this topic and Create an ApsaraDB for ClickHouse cluster.

Step 2: Migrate data to the destination ApsaraDB for ClickHouse cluster

1. Log on to the ApsaraDB for ClickHouse console.

2. On the Clusters page, click the Default Instances tab, and then click the ID of the cluster that you want to manage.

3. In the left-side navigation pane of the page that appears, click Migrate Instance.

4. On the page that appears, click Create Migration Task.

   

   Note

   If no data is migrated to the selected cluster, the message No data found. is displayed.

5. In the Select the source and destination instances step, configure Instance ID, Database Account, and Database Password in the Source Instance Information section. Then, configure Database Account and Database Password in the Destination Instance Information section.

   

6. Click Test Connectivity and Proceed to test the connectivity between the two clusters.

   • If the connection is established, proceed to the next step.

   • If the connection fails to be established, check the error message. If the message Source instance account or password error. appears, enter a valid value for the Database Account parameter or the Database Password parameter.

7. After the source cluster and destination cluster are connected, proceed to the Migration Content step. Click Next: Pre-detect and Start Synchronization.

   

   Note

   You can migrate the following objects from the source cluster: the cluster, databases, tables, data dictionaries, materialized views, user permissions, and cluster configurations.

8. After the migration task is configured, the system pre-checks the migration configuration and starts the migration task in the backend.

   • If no issues are detected during the precheck, click Completed.

   • If an issue is detected, follow the instructions to resolve the issue and migrate the data again. The following table describes the precheck items.

     Item	Description
     Instance Status Detection	Before you migrate data, make sure that no management operations, such as scale-out, upgrade, or downgrade operations, are being performed on the source ApsaraDB for ClickHouse cluster and the destination ApsaraDB for ClickHouse cluster. If management operations are being performed on the source cluster and the destination cluster, the system cannot start a migration task.
     Storage Space Detection	Before the system starts a migration task, the system checks the storage space of the source cluster and the destination cluster. Make sure that the available storage capacity of the disk for the destination cluster is greater than or equal to 1.2 times the used storage capacity of the disk for the source cluster.
     Local Table and Distributed Table Detection	If no distributed table is created for a local table or multiple distributed tables are created for the same local table, the precheck fails. You must delete redundant distributed tables to ensure that the distributed table of each local table is unique.

9. After the migration task is complete, you can view the migration task on the Migrate Instance page.

   

   Note

   If Completed is displayed in the Migration Status column, data is migrated from the source cluster to the destination cluster.

Step 3: Delete the source ApsaraDB for ClickHouse cluster