Create manual snapshots and restore data from manual snapshots - Elasticsearch

This guide demonstrates how to manually create, manage, and restore snapshots of your Elasticsearch index data on Alibaba Cloud. Snapshots are stored in a shared repository, typically an Object Storage Service (OSS) bucket.

Background

Alibaba Cloud Elasticsearch clusters utilize the elasticsearch-repository-oss plugin for data backup and restoration. This plugin is pre-installed on all Alibaba Cloud Elasticsearch clusters and cannot be removed.

For more details on the plugin, refer to its GitHub repository: elasticsearch-repository-oss.

Prerequisites

Before proceeding, ensure the following:

Object Storage Service (OSS) is configured:
- OSS must be activated in the same region as your Elasticsearch cluster.
- An OSS bucket with a Standard storage class and Public Read Access Control List (ACL) must be created.
- Refer to: Activate OSS and Create a bucket.
RAM user permissions (if applicable):
- If using a RAM user, ensure the AliyunOSSFullAccess policy is attached to it.
- Refer to: Grant permissions to a RAM user.

Usage notes

What snapshots include: Snapshots store only index data. The following cluster information is not included:
- Monitoring data (e.g., indexes starting with .monitoring or .security_audit)
- Metadata
- Translogs
- Configurations
- Software packages
- Built-in and custom plugins
- Logs
Kibana console: All code examples in this guide can be executed directly in the Kibana console of your Alibaba Cloud Elasticsearch cluster. For instructions on how to access it, see Log on to the Kibana console.

Register a snapshot repository

Before you can create snapshots, you must register a snapshot repository with your Elasticsearch cluster.

You will create a repository named my_backup.

For Alibaba Cloud Elasticsearch clusters

Use this command for clusters provisioned on Alibaba Cloud.

PUT _snapshot/my_backup/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

Parameters:

Parameter	Description
endpoint	The internal endpoint of the OSS bucket. See Regions and endpoints.
access_key_id	Your AccessKey ID. See Obtain an AccessKey pair.
secret_access_key	Your AccessKey secret. See Obtain an AccessKey pair.
bucket	The name of your OSS bucket. See Create a bucket.
compress	Enables data compression for snapshot metadata (mappings, settings). `true` (enabled) or `false` (disabled). Default is `false`.
chunk_size	The size of data parts when uploading large volumes to OSS. If a part exceeds this size, excess data is distributed to another part. Example: `"500mb"`.
base_path	The base directory within the OSS bucket for storing snapshots. Default is the root directory. Example: `"snapshot/myindex/"`.

For self-managed V8.X clusters

If you are using a self-managed Elasticsearch V8.X cluster, you must first install the elasticsearch-repository-oss plugin. See Install the elasticsearch-repository-oss plug-in.

PUT /_snapshot/my_backup
{
    "type": "oss",
    "settings": {
        "oss.client.endpoint": "oss-cn-shanghai.aliyuncs.com",
        "oss.client.access_key_id": "xxx",
        "oss.client.secret_access_key": "xxx",
        "oss.client.bucket": "xxxxxx",
        "oss.client.base_path":"snapshot/",
        "oss.client.compress": true
    }
}

Parameters:

Parameter	Description
`oss.client.endpoint`	The internal endpoint of the OSS bucket. See Regions and endpoints.
`oss.client.access_key_id`	Your AccessKey ID. See Obtain an AccessKey pair.
oss.client.secret_access_key	Your AccessKey secret. See Obtain an AccessKey pair.
`oss.client.bucket`	The name of your OSS bucket. See Create a bucket.
chunk_size	The size of data parts when uploading large volumes to OSS. If a part exceeds this size, excess data is distributed to another part. Example: `"500mb"`.
`oss.client.base_path`	The base directory within the OSS bucket for storing snapshots. Default is the root directory. Example: `"snapshot/myindex/"`.
`oss.client.compress`	Enables data compression for snapshot metadata (mappings, settings). `true` (enabled) or `false` (disabled). Default is `false`.

Query repository information

Query all repositories

GET _snapshot

Query a specific repository

GET _snapshot/my_backup

Create a snapshot

Snapshots can capture all or specific indices in your cluster. Each snapshot in a repository stores a copy of indices.

Note

The first snapshot is a full copy. Subsequent snapshots store only incremental data, making them faster to create.
A repository can store multiple snapshots.

Create a snapshot for all indices

This command creates snapshot_1 in the my_backup repository for all opened indexes.

PUT _snapshot/my_backup/snapshot_1

To block API calls until the snapshot creation is complete, use wait_for_completion=true. This is useful for scripts but can be a long wait for large indices.

PUT _snapshot/my_backup/snapshot_1?wait_for_completion=true

Create a snapshot for specific indices

By default, snapshots include all opened indices. To exclude diagnostic indices (like .kibana indices) or snapshot only specific data, specify the indices:

PUT _snapshot/my_backup/snapshot_2
{
    "indices": "index_1,index_2"
}

This command creates a snapshot containing only index_1 and index_2.

Query snapshot information

Query all snapshots in a repository

GET _snapshot/my_backup/_all

Example response:

{
  "snapshots": [
    {
      "snapshot": "snapshot_1",
      "uuid": "vIdSCkthTeGa0nSj4D****",
      "version_id": 5050399,
      "version": "5.5.3",
      "indices": [
        ".kibana"
      ],
      "state": "SUCCESS",
      "start_time": "2018-06-28T01:22:39.609Z",
      "start_time_in_millis": 1530148959609,
      "end_time": "2018-06-28T01:22:39.923Z",
      "end_time_in_millis": 1530148959923,
      "duration_in_millis": 314,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      }
    },
    {
      "snapshot": "snapshot_3",
      "uuid": "XKO_Uwz_Qu6mZrU3Am****",
      "version_id": 5050399,
      "version": "5.5.3",
      "indices": [
        ".kibana"
      ],
      "state": "SUCCESS",
      "start_time": "2018-06-28T01:25:00.764Z",
      "start_time_in_millis": 1530149100764,
      "end_time": "2018-06-28T01:25:01.482Z",
      "end_time_in_millis": 1530149101482,
      "duration_in_millis": 718,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      }
    }
  ]
}

Query a snapshot by name

GET _snapshot/my_backup/snapshot_3

Example response:

{
  "snapshots": [
    {
      "snapshot": "snapshot_3",
      "uuid": "vIdSCkthTeGa0nSj4D****",
      "version_id": 5050399,
      "version": "5.5.3",
      "indices": [
        ".kibana"
      ],
      "state": "SUCCESS",
      "start_time": "2018-06-28T01:22:39.609Z",
      "start_time_in_millis": 1530148959609,
      "end_time": "2018-06-28T01:22:39.923Z",
      "end_time_in_millis": 1530148959923,
      "duration_in_millis": 314,
      "failures": [],
      "shards": {
        "total": 1,
        "failed": 0,
        "successful": 1
      }
    }
  ]
}

Query snapshot status

The following API provides detailed information about a snapshot, including its current status and statistics for each index and shard.

GET _snapshot/my_backup/snapshot_3/_status

Example response:

{
"snapshots": [
   {
      "snapshot": "snapshot_3",
      "repository": "my_backup",
      "state": "IN_PROGRESS", 
      "shards_stats": {
         "initializing": 0,
         "started": 1, 
         "finalizing": 0,
         "done": 4,
         "failed": 0,
         "total": 5
      },
      "stats": {
         "number_of_files": 5,
         "processed_files": 5,
         "total_size_in_bytes": 1792,
         "processed_size_in_bytes": 1792,
         "start_time_in_millis": 1409663054859,
         "time_in_millis": 64
      },
      "indices": {
         "index_3": {
            "shards_stats": {
               "initializing": 0,
               "started": 0,
               "finalizing": 0,
               "done": 5,
               "failed": 0,
               "total": 5
            },
            "stats": {
               "number_of_files": 5,
               "processed_files": 5,
               "total_size_in_bytes": 1792,
               "processed_size_in_bytes": 1792,
               "start_time_in_millis": 1409663054859,
               "time_in_millis": 64
            },
            "shards": {
               "0": {
                  "stage": "DONE",
                  "stats": {
                     "number_of_files": 1,
                     "processed_files": 1,
                     "total_size_in_bytes": 514,
                     "processed_size_in_bytes": 514,
                     "start_time_in_millis": 1409663054862,
                     "time_in_millis": 22
                }
             }
          }
        }
      }
    }
  ]
}

Delete a snapshot

Use the following command to delete a snapshot. If the snapshot is currently being created, this command will stop its creation and delete it from the repository.

DELETE _snapshot/my_backup/snapshot_3

Warning

Only delete snapshots using the DELETE API. Do not manually delete snapshot data from the OSS bucket. Manually deleting snapshot files can corrupt other snapshots in the repository that share data. The DELETE API intelligently identifies and removes only the data not associated with other snapshots.

Restore indexes from a snapshot

Important

System indices: Avoid restoring system indices whose names start with a period (.) (e.g., .kibana, .monitoring). Restoring these can lead to Kibana console access failures or cluster instability.
Existing indices: If an index to be restored already exists in your target cluster, you must delete or close the existing index before starting the restoration process.
Cross-region restoration: To restore data from a snapshot to a cluster in a different region, you must first migrate the data to an OSS bucket in the destination region. Then, restore the data from that bucket to the desired Elasticsearch cluster. Refer to: Migrate data.

1. Register a repository in the target cluster

Before restoring, register a snapshot repository in your target cluster and maps the repository to the same OSS bucket where the snapshot is stored. This repository can have a different name (e.g., my_backup_restore).

PUT _snapshot/my_backup_restore/
{
    "type": "oss",
    "settings": {
        "endpoint": "http://oss-cn-hangzhou-internal.aliyuncs.com",
        "access_key_id": "xxxx",
        "secret_access_key": "xxxxxx",
        "bucket": "xxxxxx",
        "compress": true,
        "chunk_size": "500mb",
        "base_path": "snapshot/"
    }
}

2. Restore a specific index

This method is useful for verifying data or processing a single index without overwriting existing data. It renames the restored index.

POST /_snapshot/my_backup_restore/snapshot_1/_restore
{
 "indices": "index_1", 
 "rename_pattern": "index_(.+)", 
 "rename_replacement": "restored_index_$1" 
}

Parameter	Description
indices	The name of the index to restore. (e.g., `"index_1"`).
rename_pattern	A regular expression defining the pattern of the original index name.
rename_replacement	A regular expression for renaming the restored index. `"$1"` refers to the captured group from `rename_pattern`.

3. Restore all indexes, excluding system/diagnostic indices

This command restores all indices from snapshot_1, ignoring common system/diagnostic indices (.monitoring, .security, .kibana).

POST _snapshot/my_backup_restore/snapshot_1/_restore 
{"indices":"*,-.monitoring*,-.security*,-.kibana*","ignore_unavailable":"true"}

4. Restore all indexes, including system/diagnostic indices

This restores every index present in snapshot_1 to the Elasticsearch cluster.

POST _snapshot/my_backup_restore/snapshot_1/_restore

If snapshot_1 contains five indices, this command will restore all five.

Similar to snapshot creation, you can specify wait_for_completion=true to block the API call until restoration is complete:

POST _snapshot/my_backup_restore/snapshot_1/_restore?wait_for_completion=true

Query restoration information

Use the following GET API to monitor the status and progress of restoration.

Query a specific index's restoration status

GET restored_index_3/_recovery

Query restoration information for all indices

This command provides recovery details for all indices currently undergoing restoration or recently restored. Note that it may include information about shards not involved in the current restoration process.

GET /_recovery/

Example response:

{
   "restored_index_3" : {
     "shards" : [ {
       "id" : 0,
       "type" : "snapshot",
       "stage" : "index",
       "primary" : true,
       "start_time" : "2014-02-24T12:15:59.716",
       "stop_time" : 0,
       "total_time_in_millis" : 175576,
       "source" : {
         "repository" : "my_backup",
         "snapshot" : "snapshot_3",
         "index" : "restored_index_3"
       },
       "target" : {
         "id" : "ryqJ5lO5S4-lSFbGnt****",
         "hostname" : "my.fqdn",
         "ip" : "10.0.**.**",
         "name" : "my_es_node"
       },
       "index" : {
         "files" : {
           "total" : 73,
           "reused" : 0,
           "recovered" : 69,
           "percent" : "94.5%"
         },
         "bytes" : {
           "total" : 79063092,
           "reused" : 0,
           "recovered" : 68891939,
           "percent" : "87.1%"
         },
         "total_time_in_millis" : 0
       },
       "translog" : {
         "recovered" : 0,
         "total_time_in_millis" : 0
       },
       "start" : {
         "check_index_time" : 0,
         "total_time_in_millis" : 0
       }
     } ]
   }
}

The response lists all indices being restored and their shards, showing start/end times, duration, progress, and transmitted bytes.

Parameter	Description
type	The type of restoration. `snapshot` indicates restoration from a snapshot.
source	The source repository and snapshot for the shard.
percent	The restoration progress (e.g., `94.5%`).

Delete an index undergoing restoration

To cancel the restoration of an index, use the DELETE API on the target index.

DELETE /restored_index_3

This command will stop the restoration of restored_index_3 and delete any data already restored to the Elasticsearch cluster for that index.

Elasticsearch:Manual snapshots and restoration

Background

Prerequisites

Usage notes

Register a snapshot repository

Query repository information

Query all repositories

Query a specific repository

Create a snapshot

Create a snapshot for all indices

Create a snapshot for specific indices

Query snapshot information

Query all snapshots in a repository

Query a snapshot by name

Query snapshot status

Delete a snapshot

Restore indexes from a snapshot

1. Register a repository in the target cluster

2. Restore a specific index

3. Restore all indexes, excluding system/diagnostic indices

4. Restore all indexes, including system/diagnostic indices

Query restoration information

Query a specific index's restoration status

Query restoration information for all indices

Delete an index undergoing restoration

References