Technical Guide: Migrating to Alibaba Cloud Elasticsearch with Self-managed Logstash

Alibaba Cloud Elasticsearch provides a powerful solution for searching, analyzing, and visualizing your data in real-time. For businesses looking to transition their self-managed Elasticsearch environments to Alibaba Cloud, utilizing Logstash presents an effective method of migration. This comprehensive tutorial will outline the necessary steps to migrate full or incremental data seamlessly.

Precautions

Ensure that the self-managed Logstash cluster on Elastic Compute Service (ECS) instances resides in the same VPC and can connect to both Elasticsearch clusters.
Perform full data migration initially if continuous data writes occur, followed by incremental migrations based on a timestamp or unique identifiers to avoid data loss or duplication.

Step-by-Step Migration with Logstash

Step 1: Preparations

Create an Alibaba Cloud Elasticsearch cluster and deploy your self-managed clusters, including Logstash, on an ECS instance within the same VPC network.

Deploying a Self-Managed Elasticsearch Cluster:

1# Connect to ECS instance and become root user
2ssh root@your-ecs-instance-ip
3
4# Create and switch to the non-root user
5useradd elastic
6passwd elastic
7su -l elastic
8
9# Download and extract Elasticsearch
10wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.2-linux-x86_64.tar.gz
11tar -zvxf elasticsearch-7.6.2-linux-x86_64.tar.gz
12
13# Start the Elasticsearch cluster
14cd elasticsearch-7.6.2
15./bin/elasticsearch -d
16
17# Verify the cluster is running
18curl localhost:9200

Step 2: Migrate Metadata (Optional)

To maintain consistency, replicate the index metadata from the self-managed cluster to Alibaba Cloud Elasticsearch before the actual data migration.

Creating Indexes with a Python Script:

1# -*- coding: UTF-8 -*-
2
3# Python script to create indices on the destination Elasticsearch cluster
4
5import httplib
6import json
7import base64
8
9# Old and new cluster connection information
10oldClusterHost, oldClusterUserName, oldClusterPassword = "localhost:9200", "elastic", "password"
11newClusterHost, newClusterUser, newClusterPassword = "es-cn-xxxx.aliyuncs.com:9200", "elastic", "password"
12
13# Define HTTP request function
14def httpRequest(method, host, endpoint, params="", username="", password=""):
15    # Other parts of the code...
16
17# Create index manually with settings and mappings retrieved from the old cluster
18def createIndex(oldIndexName, newIndexName=""):
19    # Other parts of the code...
20
21# Execution section
22if __name__ == "__main__":
23    createIndex("index_to_migrate")

Step 3: Migrate Full Data

Configure Logstash to migrate all existing data from the self-managed cluster to Alibaba Cloud Elasticsearch.

Logstash Configuration for Full Data Migration:

# Enter the Logstash configuration directory
cd ~/logstash-7.10.0/config

# Create a new configuration file
vi es2es_all.conf

# Input the following configuration settings
input {
    elasticsearch {
        # Configuration for the source cluster
    }
}
filter {
    mutate {
        remove_field => ["@timestamp", "@version"]
    }
}
output {
    elasticsearch {
        # Configuration for the destination cluster
    }
}

# Start Logstash to begin full data migration
nohup bin/logstash -f config/es2es_all.conf > /dev/null 2>&1 &

Step 4: Migrate Incremental Data

Post full migration, configure Logstash for ongoing incremental data migrations.

# Use a similar Logstash configuration as in Step 3, but specify a query for incremental data and setup a schedule.

input {
    elasticsearch {
        # Configuration for the source cluster
        query => '{"query": {"range": {"@timestamp": {"gte": "now-5m","lte": "now/m"}}}}'
        schedule => "* * * * *"
    }
}
# Other filters and outputs remain the same.

Step 5: Validate Migration Results

Verify the migration by comparing the data on both clusters, ensuring consistency and completeness in the Alibaba Cloud Elasticsearch cluster.

# Query to check data on both source and destination clusters
GET /_cat/indices?v
POST /_search

Ready to start your journey with Elasticsearch on Alibaba Cloud? Explore our tailored Cloud solutions and services to take the first step towards transforming your data into a visual masterpiece.

Please Click here Embark on Your 30-Day Free Trial.

Community

Technical Guide: Migrating to Alibaba Cloud Elasticsearch with Self-managed Logstash

Precautions

Step-by-Step Migration with Logstash

Step 1: Preparations

Step 2: Migrate Metadata (Optional)

Step 3: Migrate Full Data

Step 4: Migrate Incremental Data

Step 5: Validate Migration Results

Read previous post:

Read next post:

Data Geek

You may also like

Comments

Data Geek

Related Products

Hologres

Cloud Migration Solution

Data Security on the Cloud Solution

Cloud Data Transfer