Elasticsearch:Create private connections for Alibaba Cloud Elasticsearch with PrivateLink and SLB

Prerequisites

• An Alibaba Cloud Elasticsearch cluster was created in VPC_1 in or after October 2020. For more information, see Create an Alibaba Cloud Elasticsearch cluster.

• An ECS instance was created in VPC_2, with the required applications deployed. For more information, see Custom launch.

Step 1: Create and configure an CLB instance

1. Log on to the CLB console.

2. Create an CLB instance that supports PrivateLink.

   1. On the Instances page, click Create CLB.

   2. On the purchase page, select the same region as the Alibaba Cloud Elasticsearch cluster, set Instance Type to Intranet, enter an instance name, and click Buy Now.

   For more information, see Create and manage a CLB instance.

3. Configure the CLB instance.

   1. On the Instances page, find the instance and click Configure Listener in the Actions column.

   2. On the Configure Server Load Balancer page, configure the listener, backend servers, and health check, and then review the configuration.

   For more information, see Configure a CLB instance and CLB listeners.

Step 2: Create an endpoint service

1. Log on to the endpoint service console.

2. In the top navigation bar, select the same region as the CLB instance.

3. Click Create Endpoint Service.

4. On the Create Endpoint Service page, set Service Resource Type to CLB, select your CLB instance for Service Resource, configure other parameters as needed, and then click OK.

For more information, see Create an endpoint service.

Step 3: Configure a private connection for the Alibaba Cloud Elasticsearch cluster

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left navigation menu, choose Elasticsearch Clusters.

3. Navigate to the target cluster.

   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.

   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. In the left navigation menu, choose Configuration and Management > Security.

5. In the Network Settings section, click Edit next to Configure Private Connection.

6. In the Configure Private Connection panel, click +Add Private Connection.

7. In the Create Private Connection dialog box, set Associated Endpoint Service and Zone, select the acknowledgment checkbox, and then click OK.

   You'll return to the Configure Private Connection panel. After the endpoint and the endpoint service are connected, the Endpoint Connection Status changes to Connected, indicating that communication is enabled between the Elasticsearch cluster in VPC_1 and client in VPC_2.

8. (Optional) View the endpoint domain name.

   You can use the endpoint domain name for other service configurations, such as Watcher, single sign-on (SSO), and LDAP.

   1. In the Configure Private Connection panel, click the target Endpoint ID.

   2. On the Endpoint Connections tab, click the icon next to the endpoint ID to view the corresponding domain name.

Prerequisites

• Two Alibaba Cloud Elasticsearch clusters are created in the same region and zone.

• The selected region and zone support PrivateLink. For supported regions, see Regions and zones that support PrivateLink.

Step 1: Create and configure an NLB instance

An NLB instance receives client requests and forwards them to backend servers based on listening rules.

1. Create an NLB instance.

   1. Log on to the NLB console.

   2. On the Instances page, click Create NLB.

   3. On the purchase page, select the same region as the Elasticsearch clusters, set the network type to Internal-facing, and select the zone.

      For more information, see Create and manage an NLB instance.

   4. Click Create Now.

2. Create an NLB server group.

   A server group routes client requests to backend servers. NLB verifies backend server availability through health checks.

   1. Go to the NLB Server Groups page.

   2. On the Server Groups page, click Create Server Group.

   3. In the dialog box, set Server Group Type to IP, enter a name, and select the VPC associated with the ES_2 cluster.

3. Add the ES_2 cluster as a backend server.

   Add the private IP address and port of the ES_2 cluster to the server group.

   1. In the Actions column for the server group, click Modify Backend Server.

   2. On the Backend Servers tab, click Add Backend Server.

   3. In the Add Backend Server wizard, select the ES_2 cluster and click Next.

   4. Specify the port. For example, reindex requires port 9200.

   5. Click OK.

4. Add a listener.

   1. Log on to the NLB console.

   2. On the Instances page, click Create Listener in the Actions column for the NLB instance.

   3. Set the listener protocol to TCP and the listener port to 9200, and then click Next.

   4. Set Server Group to IP, select the backend server group, and click Next.

   5. Click Submit.

Step 2: Create an endpoint service

1. Log on to the endpoint service console.

2. Click Create Endpoint Service.

3. Configure the endpoint service with the following settings:

   Parameter	Description
   Region	Select the same region as the Elasticsearch clusters.
   Service Resource Type	Select NLB.
   Service Resource	Selecy the NLB instance that you created.
   Service Payer	You can select either option.

   For more information, see Create and manage endpoint services.

4. Click OK.

   After the endpoint service is created, it is attached to the NLB instance and appears in the Endpoint Service section.

   

Step 3: Configure a private connection on the ES_1 cluster

Alibaba Cloud Elasticsearch automatically creates an endpoint in the network environment of the ES_1 cluster to connect to the endpoint service.

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the left navigation menu, choose Elasticsearch Clusters.

3. Navigate to the target cluster.

   1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.

   2. On the Elasticsearch Clusters page, find the cluster and click its ID.

4. In the left navigation menu, choose Configuration and Management > Security.

5. In the Network Settings section, click Edit next to Configure Private Connection.

6. In the Configure Private Connection panel, click +Add Private Connection.

7. In the dialog box, select the endpoint service you created, set Zone, and click OK.

   After the endpoint and the endpoint service are connected, the Endpoint Connection Status changes to Connected, indicating that the ES_1 and ES_2 clusters are connected.

8. (Optional) View the domain name of the endpoint.

   You can use the endpoint domain name and add it to a whitelist for service configurations such as Watcher, SSO, and LDAP.

   1. In the Configure Private Connection panel, click the ID of the destination Endpoint ID.

   2. On the Endpoint Connections tab, click the icon next to the destination endpoint ID to view the domain name that corresponds to the endpoint.

(Optional) Step 4: Verify connectivity by using reindex

You can use reindex to migrate an index from the ES_2 cluster to the ES_1 cluster to verify network connectivity.

1. Configure a reindex whitelist in the YML file of the ES_1 cluster.

   1. On the details page of the ES_1 cluster, choose Configuration and Management > Cluster Configuration.

   2. In the YML File Configuration section, click Modify Configuration in the upper-right corner and add the following entry:

      reindex.remote.whitelist: ["Domain name of the endpoint:9200"]

2. Prepare the data to migrate in the ES_2 cluster. Then, create an index in the ES_1 cluster and run the following reindex command:

   POST _reindex
   {
    "source": {
    "remote": {
    "host": "http://ep-bp1i4db71e6adaa29718-cn-hangzhou-i.epsrv-bp1fm3v8kc2qr2td6lrm.cn-hangzhou.privatelink.aliyuncs.com:9200",
    "username": "Username",
    "password": "Password"
    },
    "index": "myindex"
    },
    "dest": {
    "index": "myindex2"
    }
   }

   Category	Parameter	Description
   source	host	The access address of the PrivateLink endpoint service. The address must contain the protocol, endpoint domain name, and port number. Example: https://<Endpoint domain name>:9200.
   	username	Optional. The default username for an Alibaba Cloud Elasticsearch cluster is elastic.
   	password	The password for the username.
   	index	The name of the source index in the remote cluster.
   	query	Optional. Use a query to specify the data to migrate. For more information, see Reindex API.
   dest	index	The name of the destination index in the local cluster.

3. Verify the migration results.

   In the ES_1 cluster, check whether the data from the ES_2 cluster was successfully migrated. You can run the following command to verify:

   GET myindex2/_count

   Compare the document count with the source index in the ES_2 cluster to confirm that the migration completed successfully.