Use Curator

Install Curator

Prerequisites

• An Alibaba Cloud Elasticsearch cluster is created.

  For more information, see Create an Alibaba Cloud Elasticsearch cluster.

• An Alibaba Cloud Elastic Compute Service (ECS) instance is created, and a Python environment is prepared.

  In this example, an ECS instance that runs 64-bit CentOS 7.3 is used. The created ECS instance must reside in the same region, zone, and virtual private cloud (VPC) as the Elasticsearch cluster. For more information, see Create an instance on the Custom Launch tab.

Procedure

1. Connect to the ECS instance. For more information, see Connect to a Linux instance by using a password or key.

   Note

   In this example, a regular user is used.

2. Run the following command to install Curator:

   sudo pip install elasticsearch-curator

   Note

   • We recommend that you install Curator 5.6.0. This version supports Alibaba Cloud Elasticsearch V5.5.3 and V6.3.2 clusters. For information about the compatibility between Curator versions and Alibaba Cloud Elasticsearch versions, see Version Compatibility.

   • For more information about Curator, see Curator.

3. After Curator is installed, run the following command to check its version:

   sudo curator --version

   If the command is successfully run, the following result is returned:

   curator, version 5.6.0

Use the singleton CLI

Schedule a task by using crontab

You can use the crontab and curator commands to schedule the operations of a task.

curator [OPTIONS] ACTION_FILE
Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

Before you run the curator command, you must specify the config.yml and action.yml files.

Separate hot and cold data

For more information, see "Hot-Warm" Architecture in Elasticsearch 5.x.

Migrate indexes from hot nodes to warm nodes

1. Create a config.yml file in the /usr/curator/ directory. You can reference the following code to configure the file:

   client:
     hosts:
       - http://es-cn-0pxxxxxxxxxxxx234.elasticsearch.aliyuncs.com
     port: 9200
     url_prefix:
     use_ssl: False
     certificate:
     client_cert:
     client_key:
     ssl_no_validate: False
     http_auth: user:password
     timeout: 30
     master_only: False
   logging:
     loglevel: INFO
     logfile:
     logformat: default
     blacklist: ['elasticsearch', 'urllib3']

   • hosts: Set the value to the internal or public endpoint of the Elasticsearch cluster. The internal endpoint is used in this example.

   • http_auth: Set the value to the username and password that are used to access the Elasticsearch cluster.

2. Create an action.yml file in the /usr/curator/ directory. You can reference the following code to configure the file:

   actions:
     1:
       action: allocation
       description: "Apply shard allocation filtering rules to the specified indices"
       options:
         key: box_type
         value: warm
         allocation_type: require
         wait_for_completion: true
         timeout_override:
         continue_if_exception: false
         disable_action: false
       filters:
       - filtertype: pattern
         kind: prefix
         value: logstash-
       - filtertype: age
         source: creation_date
         direction: older
         timestring: '%Y-%m-%dT%H:%M:%S'
         unit: minutes
         unit_count: 30

   In this example, indexes that are created on hot nodes 30 minutes ago and whose names start with logstash- are migrated to warm nodes. You can also configure the action.yml file based on your business requirements.

3. Run the following command to check whether the curator command runs normally:

   sudo curator --config /usr/curator/config.yml /usr/curator/action.yml

   If the curator command runs normally, information similar to the following code is returned:

   2019-02-12 20:11:30,607 INFO      Preparing Action ID: 1, "allocation"
   2019-02-12 20:11:30,612 INFO      Trying Action ID: 1, "allocation": Apply shard allocation filtering rules to the specified indices
   2019-02-12 20:11:30,693 INFO      Updating index setting {'index.routing.allocation.require.box_type': 'warm'}
   2019-02-12 20:12:57,925 INFO      Health Check for all provided keys passed.
   2019-02-12 20:12:57,925 INFO      Action ID: 1, "allocation" completed.
   2019-02-12 20:12:57,925 INFO      Job completed.

4. Run the following command to enable the curator command to run at 15-minute intervals:

   crontab -e
   */15 * * * * curator --config /usr/curator/config.yml /usr/curator/action.yml