All Products
Search

This Product

    Elasticsearch:Configure a private connection for an Elasticsearch cluster

    Document Center

    Elasticsearch:Configure a private connection for an Elasticsearch cluster

    Last Updated:Apr 17, 2024

    Due to the limits of the new network architecture, specific features of Alibaba Cloud Elasticsearch clusters cannot be used. You can use PrivateLink and Server Load Balancer (SLB) to configure a private connection for an Alibaba Cloud Elasticsearch cluster to enable communication between the cluster and cloud resources such as an Elastic Compute Service (ECS) instance, an elastic network interface (ENI), and an elastic container instance (ECI), or between the cluster and network resources at a specific IP address. This topic describes how to use Classic Load Balancer (CLB) and PrivateLink to establish a private connection between an Alibaba Cloud Elasticsearch cluster and an Alibaba Cloud ECS instance. This topic also describes how to use Network Load Balancer (NLB) and PrivateLink to establish a private connection between Alibaba Cloud Elasticsearch clusters. You can select a solution based on the scenarios in which PrivateLink and SLB can be used.

    Note

    • Configuring private connections is the only solution that can be used to resolve the limits imposed by the new network architecture on features such as X-Pack Watcher, reindexing, Lightweight Directory Access Protocol (LDAP) authentication, and Active Directory (AD) user authentication.

    • Elasticsearch clusters that are created in October 2020 or later are deployed in the new network architecture. These Elasticsearch clusters support private connections. Elasticsearch clusters that are created before October 2020 are deployed in the original network architecture, including clusters in Alibaba Gov Cloud and Alibaba Finance Cloud. These Elasticsearch clusters do not support private connections.

    Use scenarios of PrivateLink

    Terms

    To use PrivateLink to establish private connections, you must create endpoint services and endpoints.

    Term

    Description

    endpoint service

    Endpoint services are used for establishing private connections. A VPC can use an endpoint to connect to the endpoint service in another VPC. Before a private connection can be established, you must manually create an endpoint service in the VPC that you want to access.

    endpoint

    You can associate an endpoint with an endpoint service to establish a private connection. Private connections allow a VPC to access external services. When you configure a private connection for an Elasticsearch cluster, the system automatically creates an endpoint in the network environment in which the cluster is deployed.

    Use CLB and PrivateLink to establish a private connection between an Elasticsearch cluster and an ECS instance

    Prerequisites

    • An Elasticsearch cluster is created in VPC 1 in October 2020 or later. For information about how to create an Elasticsearch cluster, see Create an Alibaba Cloud Elasticsearch cluster.

    • An ECS instance is created in VPC 2, and applications are deployed on the ECS instance. For more information, see Create an instance on the Custom Launch tab.

      Note

      • The ECS instance is used as the backend server to receive requests that are forwarded by a CLB instance. When you create an ECS instance, you must select a region and a zone that support PrivateLink. For information about the regions and zones that support PrivateLink, see Regions and zones that support PrivateLink.

      • The Elasticsearch cluster, ECS instance, and CLB instance must reside in the same zone of the same region.

    Step 1: Create and configure a CLB instance

    1. Log on to the CLB console.

    2. Create a CLB instance that supports PrivateLink.

      1. On the Instances page, click Create CLB.

      2. On the CLB (Pay-As-You-Go) International Site page, select the region in which the Elasticsearch cluster resides, set SLB Instance to Intranet, and then click Buy Now.

      For more information, see Create and manage a CLB instance.

    3. Configure the CLB instance.

      1. On the Instances page, find the CLB instance and click Configure Listener in the Actions column.

      2. On the Configure Server Load Balancer page, configure the parameters based on your business requirements and perform a health check and a configuration review.

      For more information, see Configure a CLB instance and CLB listener overview.

    Step 2: Create an endpoint service

    Note

    When you configure a private connection for the Elasticsearch cluster, the system automatically creates an endpoint in the network environment in which the cluster is deployed. You need to only create an endpoint service in the network environment in which the ECS instance is deployed.

    1. Go to the Endpoints Service page in the VPC console.

    2. In the top navigation bar, select the region in which you want to create an endpoint service.

      The endpoint service must reside in the same region as the CLB instance.

    3. Click Create Endpoint Service.

    4. On the Create Endpoint Service page, set Service Resource Type to CLB, and select the zone in which the CLB instance resides and the name of the CLB instance for the Select Service Resource parameter. Configure the remaining parameters based on your business requirements and click OK.

    For more information, see Create an endpoint service.

    Step 3: Create a private connection for the Elasticsearch cluster

    1. Log on to the Alibaba Cloud Elasticsearch console.
    2. In the left-side navigation pane, click Elasticsearch Clusters.
    3. Navigate to the desired cluster.
      1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
      2. On the Elasticsearch Clusters page, find the cluster and click its ID.

    4. In the left-side navigation pane of the page that appears, choose Configuration and Management > Security.

    5. In the Network Settings section, click Edit on the right side of Configure Private Connection.

    6. In the Configure Private Connection panel, click Add Private Connection.

    7. In the Create Private Connection dialog box, select the endpoint service that you created and select the desired zone. Then, click OK.

    8. In the Configure Private Connection panel, click Allow Connection in the Actions column for the endpoint.

      After the endpoint and the endpoint service are connected, Connected is displayed in the Endpoint Connection Status column. This indicates that communication between VPC 1 and VPC 2 is enabled.

    9. Optional. View the domain name of the endpoint.

      You can use the domain name of the endpoint for other configurations, such as X-Pack Watcher, single sign-on (SSO), and LDAP authentication.

      1. In the Configure Private Connection panel, click the ID of the endpoint in the Endpoint ID column.

      2. On the Endpoint Connections tab of the page that appears, click the 展开符 icon to the left of the endpoint ID to view the domain name of the endpoint.

    Use NLB and PrivateLink to establish a private connection between Alibaba Cloud Elasticsearch clusters

    This section describes how to add the private IP address of Elasticsearch Cluster 2 to an NLB server group to establish a private connection between Elasticsearch Cluster 1 and Elasticsearch Cluster 2. If two Elasticsearch clusters cannot communicate with each other due to the limits of the network architecture, you can use this method to establish a private connection between the clusters.

    Note

    • After a private connection is established between Elasticsearch Cluster 1 and Elasticsearch Cluster 2, you can migrate data from Elasticsearch Cluster 2 to Elasticsearch Cluster 1.

    • For more information about the features of NLB, see Functions and features.

    Prerequisites

    Elasticsearch Cluster 1 and Elasticsearch Cluster 2 that reside in the same region and zone are created.

    Note

    When you create the Elasticsearch clusters, you must select a region and a zone that support PrivateLink. For information about the regions and zones that support PrivateLink, see Regions and zones that support PrivateLink.

    Step 1: Create an NLB instance

    An NLB instance receives requests from clients and forwards requests to backend servers based on listening rules. Before you can use the NLB service, you must create an NLB instance and add a listener and a backend server in the instance.

    1. Create an NLB instance.

      1. Log on to the NLB console.

      2. On the Instances page, click Create NLB.

      3. On the NLB (Pay-As-You-Go) International Site page, configure the parameters.

        Select the region in which Elasticsearch Cluster 2 resides, set the Network Type parameter to Intranet, and then select the zone in which Elasticsearch Cluster 2 resides.

        For more information, see Create and manage an NLB instance.

    2. Create an NLB server group.

      An NLB server group forwards requests from clients to one or more backend servers specified in the server group. NLB checks the availability of backend servers by performing health checks. When you add a listener to an NLB instance, you must specify a server group. The listener checks connection requests from clients based on the configured protocol and endpoint and forwards traffic to the server group.

      1. Go to the Server Groups page in the NLB console.

      2. On the Server Groups page, click Create Server Group.

      3. In the Create Server Group dialog box, configure parameters for the server group.

        Set the Server Group Type parameter to IP, enter a name in the Server Group Name field, and then select the VPC in which Elasticsearch Cluster 2 is deployed.

    3. Add Elasticsearch Cluster 2 to the NLB instance as the backend service.

      Add the private IP address and the port number of Elasticsearch Cluster 2 to the server group that you created.

      1. Obtain the private IP address of Elasticsearch Cluster 2 by using the ping command.

        image.png

      2. In the Actions column of the server group that you created, click Modify Backend Server.

      3. On the Backend Servers tab of the page that appears, click Add IP Address.

      4. In the Add Backend Server panel, enter the private IP address of Elasticsearch Cluster 2 in the IP Address field and click Next.

      5. In the Ports/Weights step, enter the port number of Elasticsearch Cluster 2 in the Port field. For example, you can enter 9200, which is required for reindexing.

      6. Click OK.

    4. Add a listener.

      1. Log on to the NLB console.

      2. On the Instances page, find the NLB instance that you created, and click Create Listener in the Actions column.

      3. In the Configure Listener step of the Configure Server Load Balancer page, enter 9200 in the Listener Port field and click Next.

      4. In the Select Server Group step of the Configure Server Load Balancer page, select IP from the Server Type drop-down list and select the server group that you created and click Next.

      5. Click Submit.

    Step 2: Create an endpoint service

    Note

    When you configure a private connection for Elasticsearch Cluster 1, the system automatically creates an endpoint in the network environment in which the cluster is deployed. You need to only create an endpoint service in the network environment in which Elasticsearch Cluster 2 is deployed.

    1. Go to the Endpoints Service page in the VPC console.

    2. Click Create Endpoint Service.

    3. On the Create Endpoint Service page, configure the parameters based on your business requirements.

      The following table describes some parameters that you must configure when you create an endpoint service. For more information, see Create an NLB instance.

      Parameter

      Description

      Region

      Select the region in which the Elasticsearch clusters reside.

      Service Resource Type

      Select NLB.

      Select Service Resource

      Select the zone in which the Elasticsearch clusters reside and select the NLB instance that you created.

      Service Payer

      Select Service Provider or Service Consumer.

    4. Click OK.

      After the endpoint service is created, you can view that the endpoint service is associated with the NLB instance in the Service Resource section.

    Step 3: Configure a private connection for Elasticsearch Cluster 1

    When you configure a private connection for Elasticsearch Cluster 1, the system automatically creates an endpoint in the network environment in which Elasticsearch Cluster 1 is deployed.

    1. Log on to the Alibaba Cloud Elasticsearch console.
    2. In the left-side navigation pane, click Elasticsearch Clusters.
    3. Navigate to the desired cluster.
      1. In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
      2. On the Elasticsearch Clusters page, find the cluster and click its ID.

    4. In the left-side navigation pane of the page that appears, choose Configuration and Management > Security.

    5. In the Network Settings section, click Edit on the right side of Configure Private Connection.

    6. In the Configure Private Connection panel, click Add Private Connection.

    7. In the Create Private Connection dialog box, select the endpoint service that you created and select the desired zone. Then, click OK.

    8. In the Configure Private Connection panel, click Allow Connection in the Actions column for the endpoint.

      After the endpoint and the endpoint service are connected, Connected is displayed in the Endpoint Connection Status column. This indicates that communication between Elasticsearch Cluster 1 and Elasticsearch Cluster 2 is enabled.

    9. Optional. View the domain name of the endpoint.

      You can obtain the domain name of the endpoint and add the domain name to a whitelist for other configurations, such as X-Pack Watcher, SSO, and LDAP authentication.

      1. In the Configure Private Connection panel, click the ID of the endpoint in the Endpoint ID column.

      2. On the Endpoint Connections tab of the page that appears, click the 展开符 icon to the left of the endpoint ID to view the domain name of the endpoint.

    Step 4: (Optional) Use the reindex API to test the connectivity between the Elasticsearch clusters

    Use the reindex API to migrate indexes in Elasticsearch Cluster 2 to Elasticsearch Cluster 1 to test the network connectivity between the clusters.

    1. Configure a reindex whitelist in the YML configuration file in Elasticsearch Cluster 1.

      1. In the left-side navigation pane of the details page of Elasticsearch Cluster 1, choose Configuration and Management > Cluster Configuration.

      2. In the YML Configuration section of the page that appears, click Modify Configuration. 

        reindex.remote.whitelist: ["Domain name of the endpoint:9200"]

    2. Prepare data that you want to migrate in Elasticsearch Cluster 2, create an index in Elasticsearch Cluster 1, and then use the reindex API to migrate the data to Elasticsearch Cluster 1. 

      POST _reindex
{
 "source": {
 "remote": {
 "host": "http://ep-bp1i4db71e6adaa29718-cn-hangzhou-i.epsrv-bp1fm3v8kc2qr2td6lrm.cn-hangzhou.privatelink.aliyuncs.com:9200",
 "username": "Username",
 "password": "Password"
 },
 "index": "myindex"
 },
 "dest": {
 "index": "myindex2"
 }
}

      Category

      Parameter

      Description

      source

      host

      The URL of the endpoint service. The URL must contain the protocol, the domain name of the endpoint, and the port number. You must configure this parameter in the https://Domain name of the endpoint:9200 format.

      username

      Optional. The username that is used to connect to the remote cluster. The default username is elastic.

      password

      The password that is used to connect to the remote cluster. The password is specified when you create the cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting a password, see Reset the access password for an Elasticsearch cluster.

      index

      The source index in the remote cluster.

      query

      Specifies the data that you want to migrate. For more information, see Reindex API.

      dest

      index

      The destination index in the local cluster.

    3. Check whether data in Elasticsearch Cluster 2 is migrated to Elasticsearch Cluster 1.

    References