To remove nodes that you no longer need, you must perform the operation on the Node Pools page in the Container Service for Kubernetes (ACK) console. This helps prevent user errrors. When you remove a node, the pods that run on the node are migrated to other nodes. This may cause service interruptions. We recommend that you remove nodes during off-peak hours.
Prerequisites
An ACK cluster is created. For more information, see Create an ACK managed cluster or Create an ACK dedicated cluster.
A kubectl client is connected to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.
How to remove a node
After you remove nodes from a node pool, ACK automatically adjusts the expected number of nodes in the node pool. The new expected number of nodes equals the original expected number of nodes minus the number of nodes that are removed. For more information about how to modify the expected number of nodes, see Scale a node pool.
When you remove a node, you can select the following options:
Release ECS Instance: Release the Elastic Compute Service (ECS) instance after the node is removed.
ImportantIf the ECS instance uses the subscription billing method, the system does not release the instance after the node is removed.
Drain Node: Automatically drain the node before the node is removed.
NoteThe timeout period of a node removal task is 30 minutes. If the terminationGracePeriodSeconds parameter (graceful shutdown period) of the pods on a node is set to a large value, such as 40 minutes, node draining cannot be completed before the node removal task times out. This is because the timeout period of the node removal task takes precedence over the graceful shutdown period and the graceful shutdown period is longer than the timeout period. In this case, node draining will be terminated after 30 minutes and considered failed. You need to manually drain the node.
When node draining fails, the node removal task is immediately canceled. If your application pods on the node require a longer graceful shutdown period, you can run the
kubectl drain [node-name] [options]command to manually evict the pods from the node. After the node is drained, you can then remove the node. In this case, do not select the Drain Node option when you remove the node.
The system performs the following steps to remove a node from a node pool:
If you select the Drain Node option, the system first drains the node. If the system fails to drain the node, the node removal task is canceled. Make sure that the cluster has sufficient node resources for the pods that are evicted from the node that you want to remove. You can run the
kubectl drain [node-name] [options]command to manually drain the node before you remove the node. In this case, do not select the Drain Node option when you remove the node.Remove the node from the Kubernetes API server of the cluster.
Remove the node from the scaling group. If the ECS instance uses the subscription billing method, the system does not release the instance even if you select the Release ECS Instance option. You can remove the instance in the ECS console or by calling the ECS API. If you select the Release ECS Instance option and the ECS instance uses the pay-as-you-go billing method, the system releases the instance after the node is removed. We recommend that you back up the data on the node before you remove the node.
Usage notes
When you remove a node, the pods that run on the node are migrated to other nodes. This may cause service interruptions. We recommend that you remove nodes during off-peak hours.
Unknown errors may occur when you remove nodes. Before you remove nodes, back up the data on the nodes.
Nodes remain in the Unschedulable state when they are being removed.
You can remove only worker nodes. You cannot remove master nodes.
We recommend that you remove nodes in the ACK console. If you run the kubectl delete node command to remove nodes from an ACK cluster, take note of the following limits:
The removed nodes cannot be added to other ACK clusters.
The Elastic Compute Service (ECS) instance on which the node is deployed is released after the node is removed.
If you use the ECS or Auto Scaling console to remove nodes or release instances from a node pool, the nodes or instances are automatically suspended and then removed from the ACK console. The same rule applies if you use the ECS or Auto Scaling API to remove nodes or release instances, or the subscription of the instances in the node pool expires. If the node pool is configured with an expected number of nodes, the node pool automatically scales to the expected number of nodes based on the node pool configurations.
Procedure
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Node Pools page, click the name of the node pool that you want to manage. On the Nodes tab, select the nodes that you want to remove and click Batch Remove in the lower part of the page.
Optional: In the Remove Node dialog box, you can select Release ECS Instance and Drain Node.
Release ECS Instance:
Only pas-as-you-go ECS instances and the data disks on the instances are released. You are still charged for the ECS instances that are not released.
Subscription ECS instances are automatically released after the subscriptions expire.
If you do not select Release ECS Instance, you are still charged for the ECS instance where the node is deployed.
Drain Node: Migrates pods that run on the nodes to be removed to other nodes in the cluster. If you select this option, make sure that other nodes in the cluster have sufficient resources to host these pods.
You can also run the
kubectl drain node-name [options]command to migrate pods that run on the nodes to other nodes in the cluster.Notenode-name must be in the your-region-name.node-id format.
your-region-name specifies the region where the cluster that you want to manage is deployed. node-id specifies the ID of the ECS instance where the node to be removed is deployed. Example of node-name: cn-hangzhou.i-i-bp1asavedmte377c3****.
optionsspecifies the optional parameters of the command. Example of options:--force --ignore-daemonsets --delete-local-data. You can run thekubectl drain --helpcommand to view help information.
Select I understand the above information and want to remove the node(s) and click OK.
NoteIf a message that indicates a node removal failure appears, clear Release ECS Instance and Drain Node, and then remove the nodes again.
FAQ
How can I remove a node if I cannot find the node on the Nodes tab of the node pool?
Perform the following operations to remove the node:
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, click the name of the cluster that you want to manage and choose in the left-side navigation pane.
On the Nodes page, select the nodes that you want to remove and click Batch Remove in the lower part of the page.
In the Remove Node dialog box, confirm the information and click OK.
References
ACK allows you to scale a node pool by modifying the expected number of nodes in the node pool. You can scale out node pools to meet the requirements of business development and scale in node pools to reduce resource costs. For more information, see Scale a node pool.
You can add existing ECS instances to an ACK cluster or re-add existing nodes to a node pool. To do this, you can log on to the ACK console and manually add nodes or configure ACK to automatically add nodes. For more information, see Add existing ECS instances to an ACK cluster.
For information about how to update a node pool without affecting data disks, see Node pool updates.