Elasticsearch:Use a self-managed Logstash cluster to migrate data to an Alibaba Cloud Elasticsearch cluster

Step 1: (Optional) Create a role and a user

You need to use an Elasticsearch user when you connect a self-managed Logstash cluster to an Alibaba Cloud Elasticsearch cluster or view Logstash monitoring data in the Kibana console. You can use the elastic user or a custom user.

Step 2: Install Logstash on the ECS instance

1. Connect to the ECS instance.

   For more information, see Connect to an ECS instance.

2. Download the Logstash installation package.

   Visit the Elastic website and download the Logstash installation package that has the same version as the Elasticsearch cluster.

   wget https://artifacts.elastic.co/downloads/logstash/logstash-8.5.1-linux-x86_64.tar.gz

   Note

   Command for Logstash V7.4:

   wget https://artifacts.elastic.co/downloads/logstash/logstash-7.4.0.tar.gz

3. Decompress the Logstash installation package.

   In this example, Logstash V8.5.1 is used.

   tar -xzvf logstash-8.5.1-linux-x86_64.tar.gz

Step 3: Configure and run a Logstash pipeline

1. Go to the Logstash directory.

   cd logstash-8.5.1

2. Create a .conf file.

   touch test.conf

3. Configure the test.conf file.

   The following sample code provides a configuration example:

   input {
       file {
           path => "/your/file/path/xxx"
           }
   }
   filter {
   }
   output {
     elasticsearch {
       hosts => "<http://<instanceId>.elasticsearch.aliyuncs.com:9200>"
     	index => "<index-name>"
       user => "<user-name>"
       password => "<logstash-password>"
     }
   }

   Parameter	Description
   path	The path of the log file. In this example, /var/log/messages is used.
   hosts	The endpoint of the Elasticsearch cluster. Replace instanceId with the ID of the Elasticsearch cluster. You can query the cluster ID on the Basic Information page of the cluster. Example: http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200.
   index	The name of the index to which the data in the log file is written. You can specify a custom name.
   user	The username that is used to access the Elasticsearch cluster. In this example, elastic is used. Important To ensure system security, we recommend that you use a custom user instead of the elastic user in actual business. The username must be enclosed in a pair of double quotation marks ("). This avoids potential errors caused by special characters in the username when you run the Logstash pipeline.
   password	The password that is used to access the Elasticsearch cluster. Important The password must be enclosed in a pair of double quotation marks ("). This avoids potential errors caused by special characters in the password when you run the Logstash pipeline.

   Logstash provides a variety of input, filter, and output plug-ins. You need only to simply configure these plug-ins for data transfer. For more information, see Structure of a Config File.

4. Run the Logstash pipeline.

   bin/logstash -f test.conf

   If the command is successfully run, Logstash automatically detects changes to the log file and synchronizes the changes to the Elasticsearch cluster. Logstash automatically synchronizes data to the Elasticsearch cluster whenever data is added to the monitored log file.

Step 4: View the migrated data

1. Log on to the Kibana console of the Elasticsearch cluster.

   For more information, see Log on to the Kibana console.

2. Click the icon in the upper-left corner of the page that appears, and choose Management > Dev Tools.

3. On the Console tab of the page that appears, run the following command:

   GET /logstash-*/_search

Step 5: (Optional) Monitor the Logstash node

1. Navigate to the config folder in the Logstash directory.

   cd logstash-8.5.1/config

2. Configure the logstash.yml file.

   1. Open the logstash.yml file.

      vim logstash.yml

   2. In the logstash.yml file, delete the annotations of the parameters that are listed in the following table. Then, configure the parameters.

      xpack.monitoring.enabled:true
      xpack.monitoring.elasticsearch.username:"<user-name>"
      xpack.monitoring.elasticsearch.password:"<logstash-password>"
      xpack.monitoring.elasticsearch.hosts:["http://es-cn-nwy3fwjp8xxxxxxx.elasticsearch.aliyuncs.com:9200"]

      Parameter	Description
      xpack.monitoring.enable	Set this parameter to true. Default value: false.
      xpack.monitoring.elasticsearch.username	The name of the user that you created for Logstash monitoring. In this example, the elastic user is used. Important To ensure system security, we recommend that you use a custom user instead of the elastic user in actual business.
      xpack.monitoring.elasticsearch.password	The password of the user that you created for Logstash monitoring.
      xpack.monitoring.elasticsearch.hosts	The endpoint of the Elasticsearch cluster. Example: http://es-cn-45xxxxxxxxxxxxju.elasticsearch.aliyuncs.com:9200.

3. Go to the Logstash directory and launch the Logstash service.

   cd ../
   bin/logstash -f test.conf

4. Log on to the Kibana console of the Elasticsearch cluster.

   For more information, see Log on to the Kibana console.

5. Click the icon in the upper-left corner of the page that appears, and choose Management > Stack Monitoring.

6. (Optional) In the Create rules dialog box, click OK.

7. On the Cluster overview page, view Logstash monitoring data.