Elasticsearch:Use the reindex API to migrate data between Alibaba Cloud Elasticsearch clusters

Scenarios

This section describes the use scenarios of the reindex API in data migration across Alibaba Cloud Elasticsearch clusters. You can select a solution based on your business data and the network architecture in which your Elasticsearch clusters are deployed.

• You want to migrate data between Elasticsearch clusters.

• The shards for an index in an Elasticsearch cluster are inappropriately distributed. For example, the volume of data stored in an index is excessively large, but the number of shards for the index is excessively small. In this case, you can use the reindex API to reindex data.

• You want to modify mappings for an index that stores a large volume of data. In this case, you can use the reindex API to copy the data in the index.

Prerequisites

• Two Alibaba Cloud Elasticsearch clusters that are deployed in the original network architecture, the same virtual private cloud (VPC), and the same vSwitch are created. In this example, an Elasticsearch V6.7.0 cluster is used as the local cluster, and an Elasticsearch V6.3.2 cluster is used as the remote cluster.

• Test data is prepared.

  • Local cluster

    Create a destination index in the local cluster.

    PUT dest
    {
      "settings": {
        "number_of_shards": 5,
        "number_of_replicas": 1
      }
    }

  • Remote cluster

    Prepare the data that you want to migrate in the remote cluster. In this example, the data in the "Getting started" topic is used. For more information, see Getting started.

    Important

    If you want to use an Elasticsearch cluster of V7.0 or later as a remote cluster, you must set the index type to _doc.

Procedure

1. Log on to the Alibaba Cloud Elasticsearch console.
2. In the left-side navigation pane, click Elasticsearch Clusters.
3. Navigate to the desired cluster.
   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. Configure a reindex whitelist for the local cluster.

   1. In the left-side navigation pane of the page that appears, choose Configuration and Management > Cluster Configuration.

   2. On the page that appears, click Modify Configuration on the right side of YML Configuration.

   3. In the YML File Configuration panel, enter the endpoint and port number of the remote cluster in the Other Configurations field.

      The configurations of the reindex whitelist vary based on the number of zones of the remote cluster.

      • If the remote cluster is a single-zone cluster, configure the reindex whitelist in the <Endpoint of the cluster>:9200 format. Sample code:

        reindex.remote.whitelist: ["es-cn-09k1rgid9000g****.elasticsearch.aliyuncs.com:9200"]

      • If the remote cluster is a multi-zone cluster, the reindex whitelist must contain the IP addresses of all data nodes in the cluster and the port number of the cluster. Sample code:

        reindex.remote.whitelist: ["10.0.xx.xx:9200","10.0.xx.xx:9200","10.0.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200","10.15.xx.xx:9200"]

        Note

        You can obtain the IP addresses of all data nodes in a cluster from the Node Visualization tab on the Basic Information page of the cluster. For more information, see View the cluster status and node information.

      For information about the configuration of a reindex whitelist, see Configure a remote reindex whitelist.

   4. Select This operation will restart the cluster. Continue? and click OK.

      Then, the system restarts the Elasticsearch cluster. You can view the restart progress in the Tasks dialog box. After the cluster is restarted, the configuration is complete.

5. In the local cluster, call the reindex API to reindex data.

   1. Log on to the Kibana console of the local cluster and go to the homepage of the Kibana console as prompted.

      For more information about how to log on to the Kibana console, see Log on to the Kibana console.

      Note In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.

   2. In the left-side navigation pane of the page that appears, click Dev Tools.

   3. On the Console tab, run the following command to call the reindex API to reindex data:

      POST _reindex
      {
        "source": {
          "remote": {
            "host": "http://es-cn-09k1rgid9000g****.elasticsearch.aliyuncs.com:9200",
            "username": "elastic",
            "password": "your_password"
          },
          "index": "product_info",
          "query": {
            "match": {
              "productName": "Wealth management"
            }
          }
        },
        "dest": {
          "index": "dest"
        }
      }

      Category	Parameter	Description
      source	host	The URL that is used to connect to the remote cluster. The URL must contain the protocol, endpoint, and port number. Example: https://otherhost:9200. The value of the host parameter varies based on the number of zones of the remote cluster. If the remote cluster is a single-zone cluster, you must configure this parameter in the http://<Endpoint of the remote cluster>:9200 format. Note You can obtain the endpoint on the Basic Information page of the remote cluster. For more information, see View the basic information of a cluster. If the remote cluster is a multi-zone cluster, you must configure this parameter in the http://<IP address of a data node in the remote cluster>:9200 format.
      	username	The username that is used to connect to the remote cluster. This parameter is required only if basic authentication needs to be performed on requests that are sent to the remote cluster. The default username that is used to connect to Alibaba Cloud Elasticsearch clusters is elastic. Important For security purposes, we recommend that you use HTTPS to send requests if basic authentication needs to be performed. Otherwise, the required password is transmitted in plaintext. For Alibaba Cloud Elasticsearch clusters, you can use HTTPS in host only after you enable the protocol for the clusters. For information about how to enable HTTPS for an Elasticsearch cluster, see Enable HTTPS.
      	password	The password that is used to connect to the remote cluster. The password is specified when you create the cluster. If you forget the password, you can reset it. For information about the procedure and precautions for resetting a password, see Reset the access password for an Elasticsearch cluster.
      	index	The source index in the remote cluster.
      	query	Specifies the data that you want to migrate. For more information, see Reindex API.
      dest	index	The destination index in the local cluster.

      Note

      When you reindex data from a remote cluster, manual slicing and automatic slicing are not supported for the data. For more information, see Manual slicing and Automatic slicing.

      If the command is successfully run, the following result is returned:

      {
        "took" : 51,
        "timed_out" : false,
        "total" : 2,
        "updated" : 2,
        "created" : 0,
        "deleted" : 0,
        "batches" : 1,
        "version_conflicts" : 0,
        "noops" : 0,
        "retries" : {
          "bulk" : 0,
          "search" : 0
        },
        "throttled_millis" : 0,
        "requests_per_second" : -1.0,
        "throttled_until_millis" : 0,
        "failures" : [ ]
      }

6. Run the following command to view the migrated data:

   GET dest/_search

   If the command is successfully run, the following result is returned:

   • Single-zone cluster

   • Multi-zone cluster

Summary

Additional information