If you want to modify the configurations of an Elasticsearch cluster or node, or an Elasticsearch cluster or node encounters an exception, you may need to restart the Elasticsearch cluster or node. This topic describes how to restart an Elasticsearch cluster or node in the Elasticsearch console.
Prerequisites
The following operations are performed:
Make sure that the cluster is in the Normal state (indicated by the color green), each index in the cluster has at least one replica shard, and the resource usage of each node in the cluster is not high.
ImportantBefore you make a change to a multi-zone cluster, you must make sure that the number of replica shards of each index in the cluster is less than the number of zones in which the cluster is deployed. After the change is complete, you can manually increase the number of replica shards based on your business requirements. For information about how to change the number of replica shards of indexes in a cluster, see Index Templates.
To view the number of replica shards for the indexes in the cluster, log on to the Kibana console of the cluster, run the
GET _cat/indices?vcommand, and then view the value of the rep parameter for each index. If the value of the parameter for each index is greater than or equal to 1, each index has at least one replica shard. For information about how to log on to the Kibana console, see Log on to the Kibana console.To view the resource usage of each node in the cluster, go to the Cluster Monitoring page. For example, the value of NodeCPUUtilization(%) is about 80%, that of NodeHeapMemoryUtilization(%) is about 50%, and that of NodeLoad_1m(value) is less than the number of vCPUs for the node. For more information, see Metrics and exception handling suggestions.
In the Kibana console of your cluster, check whether your cluster stores indexes in the close state. If your cluster stores such indexes, you must open the indexes. Otherwise, the upgrade fails.
Run the following command to view the statuses of indexes:
GET /_cat/indices?v
Run the following command to open an index in the close state:
POST /<index_name>/_openReplace <index_name> with the name of the index in the close state.
Precautions
In most cases, if the load of a cluster is not high and the indexes in the cluster have replica shards, the cluster can still provide services during a restart. However, access timeouts may occur during a restart in some cases. For example, if a number of nodes in the cluster are forced to restart at the same time, the cluster is heavily loaded and is not accessible, the indexes in the cluster do not have replica shards, or large amounts of data are written or queried during a restart or forced restart, access timeouts may occur. In these cases, we recommend that you design a retry mechanism on your client first and restart the cluster during off-peak hours.
Procedure
- Log on to the Alibaba Cloud Elasticsearch console.
- In the left-side navigation pane, click Elasticsearch Clusters.
- Navigate to the desired cluster.
- In the top navigation bar, select the resource group to which the cluster belongs and the region where the cluster resides.
- On the Elasticsearch Clusters page, find the cluster and click its ID.
In the upper-right corner of the Basic Information page, click Restart.
In the Restart dialog box, configure the parameters.
Parameter
Description
Object
The type of the object that you want to restart. Valid values:
Cluster: indicates all nodes in the cluster.
Node: indicates specific nodes in the cluster.
Node Role: indicates specific types of nodes in the cluster.
Nodes
The nodes or types of nodes that you want to restart. This parameter is displayed only if you set the Object parameter to Node or Node Role.
If you set the Object parameter to Node, select one or more nodes that you want to restart.
If you set the Object parameter to Node Role, select the type of the node that you want to restart, such as Data Node or Kibana Node. You can select multiple types.
Blue-green Update
Specifies whether to perform a blue-green restart. This parameter is displayed only if you set the Object parameter to Node or Node Role.
If you select Blue-green Update, the system adds a node to your cluster, migrates the data on the original node to the new node, and removes the original node during the restart. If the performance of a node in your cluster does not meet your requirements, you can use the Blue-green Update feature. For example, if the CPU utilization of the node remains high for a long period of time, you can use this feature.
ImportantIf you want to use the Blue-green Update feature, make sure that your cluster is in the Normal state. In addition, you must set the Restart Mode parameter to Restart.
If you select Blue-green Update, the IP address of the node that you want to restart changes after the restart. If you have specified the IP address of the node in the configuration file of your cluster, you must update the IP address of the node in the configuration file after the restart.
Clusters with the specifications of 1 vCPU and 2 GiB of memory do not support the Blue-green Update feature. If you want to use the Blue-green Update feature for such a cluster, you must first upgrade the specifications of the cluster to 2 vCPUs and 4 GiB of memory or higher. For more information, see Upgrade the configuration of a cluster.
Restart Mode
The restart mode. Valid values:
Restart: You can use this mode only if the cluster is in the Normal state. Otherwise, you must use the Forced Restart mode. If your cluster is in the Normal state and you select the Restart mode, your cluster can still provide services during the restart, but the restart is time-consuming.
ImportantDuring the restart of a node, the CPU utilization and memory usage of the node surge. This may affect the service stability of your cluster for a short period of time.
The time that is required to restart a cluster varies based on multiple factors, such as the volume of data stored in the cluster. You can view the restart progress in the Tasks dialog box.
Forced Restart: If your cluster is in an abnormal state (indicated by the color yellow or red), you can use only this mode to restart the cluster or nodes in the cluster.
ImportantIf the disk usage exceeds the value of cluster.routing.allocation.disk.watermark.low, your cluster may be in an abnormal state. In this case, we recommend that you do not perform the following operations on the cluster: scale-out, disk capacity expansion, restart, password reset, and other operations that may change the configuration of the cluster. Perform the preceding operations only after the state of the cluster becomes Normal.
Concurrency
The number of nodes that are restarted at the same time. The default value is calculated by using the following formula:
(1/Total number of nodes in the cluster) × 100. Unit: %. For example, if the concurrency is 10%, 10% of nodes in the cluster are restarted at the same time.This parameter is displayed only if you set the Restart Mode parameter to Forced Restart.
You can increase the concurrency to accelerate a forced restart. The higher the concurrency, the faster a forced restart, but the greater the impact on the cluster. If the concurrency is 100%, all nodes will be restarted at the same time. As a result, the cluster cannot be accessed, and the cached data that has not been written to disks may be lost. We recommend that you set the concurrency to 100% in scenarios where your cluster encounters an exception and needs to be recovered at the earliest opportunity.
NoteThe value of Estimated to Take is calculated by multiplying the average time of previous node restarts by the total number of nodes. The actual restart time prevails.
Click OK.
NoteIf you select the Forced Restart mode, you must select Restart Cluster Forcibly to confirm the restart.
During the restart, the value of Status for the cluster is Initializing. You can click
in the upper-right corner to view the details in the Tasks dialog box. After the cluster is restarted, the value of Status is Normal.