Elasticsearch:Migrate Elasticsearch nodes in a zone

Background information

• Zone: In most cases, a data center corresponds to a zone. In some cases, a data center corresponds to multiple zones. For example, the AM5 data center corresponds to the cn-hangzhou-d and cn-hangzhou-finance-d zones. To facilitate Elasticsearch management, if a data center is removed, the related zones are removed.

• Node migration across zones: Elasticsearch provides standardized capabilities to allow you to migrate nodes for a cluster across zones. The migration is implemented based on the blue-green deployment capability of Elasticsearch. This enables smooth, graceful cluster changes.

Prerequisites

The following operations are performed:

• Make sure that your Elasticsearch cluster is in the Normal state.

  You can run the GET _cat/health?v command to view the status of the cluster.

• In the Kibana console of your cluster, check whether your cluster stores indexes in the close state. If your cluster stores such indexes, you must open the indexes. Otherwise, the upgrade fails.

  • Run the following command to view the statuses of indexes:

    GET /_cat/indices?v

    

  • Run the following command to open an index in the close state:

    POST /<index_name>/_open

    Replace <index_name> with the name of the index in the close state.

• Make sure that a zone with sufficient resources exists within the current account.

  We recommend that you select the destination zone from bottom to top in alphabetical order because the zone may have sufficient resources. For example, if both cn-hangzhou-e and cn-hangzhou-h are available, select cn-hangzhou-h. After you migrate nodes to another zone, you must manually upgrade the configuration of your Elasticsearch cluster. For more information, see Upgrade the configuration of a cluster.

Precautions

• You can migrate the nodes only in a single zone to another zone at a time. If you want to migrate the nodes in multiple zones, you need to perform migration multiple times.

• After you migrate the nodes in a zone to another zone, the system performs a rolling restart for your cluster. The time required for the migration depends on the size, data volume, and load of your cluster. We recommend that you perform migration during off-peak hours.

• In most cases, if the indexes of your cluster have replica shards and the load of your cluster is normal, your cluster can still provide services during migration. The following items indicate that the load of a cluster is normal: The CPU utilization of each node in the cluster is about 60%, the heap memory usage of each node in the cluster is about 50%, and the value of NodeLoad_1m for each node is less than the number of vCPUs for the node.

• If the indexes of your cluster do not have replica shards, the load of the cluster is excessively high, and large amounts of data are written to or queried in your cluster, access timeouts may occur during migration. We recommend that you configure an access retry mechanism for your client before you migrate nodes in a zone. This reduces the impact on your business.

Procedure

1. Log on to the Alibaba Cloud Elasticsearch console.
2. In the left-side navigation pane, click Elasticsearch Clusters.
3. Navigate to the desired cluster.
   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. In the Node Visualization section of the Basic Information page, move the pointer over the zone whose nodes you want to migrate. Then, click Migrate.

   

5. In the Migrate Nodes dialog box, configure Destination Zone and vSwitch.

   Parameter	Description
   Destination Zone	Select the zone to which you want to migrate nodes. The selected zone may also have insufficient resources or may be unavailable. In this case, select another zone. To ensure high availability of nodes, you cannot select other zones where nodes for the Elasticsearch cluster are deployed.
   vSwitch	If your cluster is a single-zone cluster, you must select a new vSwitch for node migration. In most cases, the vSwitch that is provided by default is used. Note If your cluster is a multi-zone cluster or a cluster deployed on the Alibaba Finance Cloud, you do not need to select a new vSwitch.

   Important

   Node migration triggers a cluster restart. During the restart, the cluster can still provide services, but the services may be unstable. Therefore, we recommend that you migrate nodes during off-peak hours.

6. Read the terms of data migration, select the check box, and then click OK.

   Then, the system restarts the cluster. After the cluster is restarted, the nodes are migrated to the destination zone.

   Important

   • During the migration, you need to purchase dedicated master nodes in the destination zone. In this case, the original zone and destination zone coexist.

   • After the migration, the original zone may still be displayed on the Basic Information and Upgrade/Downgrade pages in the Elasticsearch console due to delayed information updates. This does not affect the use of the cluster in the new zone. You can view the actual zones where the cluster resides in the Node Visualization section of the Basic Information page.

   • After the nodes in the zone are migrated, the IP addresses of the nodes change. If you specified the IP addresses in the configuration of your cluster, you must update the IP addresses after the migration.