If you need to remove a worker node from a node pool or cluster, you can do so through the Container Service Management Console. Consult the precautions and feature descriptions in this topic and carry out the process during non-peak business hours for a smooth operation.
Scenarios
-
For clusters created before the node pool feature, you may have standalone worker nodes. To facilitate batch operations, you might want to remove a node from the cluster and later add it to an existing node pool.
-
When upgrading to a new ECS instance, you may want to replace an old instance in the node pool with the new one, necessitating the removal of the old instance.
-
If your node pool has manual or automatic scale-out enabled, you may need to scale in a specific ECS instance to reduce costs.
-
To migrate nodes between node pools, remove the node from the original pool and add it to the new one.
-
If an ECS instance requires an upgrade or downgrade, remove the node and re-add it to the node pool after making changes in the ECS console.
-
When changing the billing type of an ECS instance, such as from pay-as-you-go to subscription, remove the node and re-add it to the node pool after the change.
-
If a node is behaving abnormally, you can reinitialize it by removing and then re-adding it to the cluster.
This operation will reset the node by replacing the system disk. Ensure you back up any important data on the system disk beforehand.
Precautions
-
When removing a node, perform standardized operations in the Container Service Management Console. Do not manually remove nodes using
kubectl delete node. -
Do not release nodes directly in the ECS or ESS console, or through related APIs. Also, avoid passive node release due to subscription expiration. In such cases, the node will be automatically removed from the Container Service Management Console.
If the node pool has a set number of expected nodes, it will automatically scale out other instances to maintain this number.
-
Node removal involves pod migration, which can impact business operations. Schedule this during off-peak hours and back up data beforehand to mitigate risks.
-
ACK will perform a drain operation when removing a node. Ensure there are enough resources on other nodes to prevent application pods from becoming unschedulable.
-
Review the node affinity rules and scheduling policies of pods on the node to be removed to ensure they remain schedulable on other nodes.
Feature description
Removing a node means taking one or more worker nodes out of a node pool or cluster. If your node pool has an Expected Number Of Nodes set, it will automatically adjust to maintain this number.
If you plan to re-add the instance to the cluster after removal, see automatically or manually add nodes.
Before removal, decide whether to automatically drain the node (safely evict pods to other nodes) and whether to release the ECS instance and its data disk simultaneously.
Automatically Drain The Node (drain)
Perform a drain operation before removal to safely evict pods to other nodes and prevent service interruption. Ensure other nodes have sufficient resources to avoid unschedulable application pods. Note the following:
-
The system's default pod graceful termination timeout is 30 minutes. If a pod has a longer configured terminationGracePeriodSeconds, the system will still enforce the 30-minute limit. After this period, if the pod hasn't terminated gracefully, the system will forcibly stop the drain and halt the node removal process. You will need to manually retry.
If your application pod requires a longer graceful termination time, manually drain the node using
kubectl drain <nodeName> [options]before eviction. Once there are no application pods on the node, uncheck Automatically Drain The Node (drain) and proceed with node removal. -
Pod Disruption Budget (PDB) configuration involves ensuring that the node drain operation, which evicts pods from the node to other active nodes, complies with the PDB. Verify and modify or remove PDB rules as necessary to prevent issues during the draining process.
Release ECS At The Same Time
After node removal, you can opt to release the ECS instance and its data disk. Always back up your data first. If not selected, the ECS instance will continue to incur charges.
-
Only pay-as-you-go ECS instances and their data disks are eligible for release. Instances not released will continue to be billed.
-
Subscription ECS instances are set to release automatically once their billing cycle concludes. You can request a refund to terminate the instance before its expiration, or switch the subscription to a pay-as-you-go model before releasing it.
Procedure
Remove from a node pool
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose .
-
Click the target node pool name, select the target node on the Node Management page, and click Batch Remove at the bottom of the page.
-
Decide if you need Automatically Drain The Node (drain) and Release ECS At The Same Time, read the precautions, and follow the prompts to complete the operation.
For details on Automatically Drain The Node (drain) and Release ECS At The Same Time, refer to the feature description above.
Remove from a cluster
Log on to the ACK console. In the left-side navigation pane, click Clusters.
On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose .
-
Select the target node and click Batch Remove at the bottom of the page.
-
Decide if you need Automatically Drain The Node (drain) and Release ECS At The Same Time, read the precautions, and follow the prompts to complete the operation.
For details on Automatically Drain The Node (drain) and Release ECS At The Same Time, refer to the feature description above.
FAQ
How to handle node removal failure?
Node removal may fail due to issues like graceful termination time or PDB configuration. Refer to the feature description for guidance. In such cases, manually drain the node using kubectl drain <nodeName> [options] before eviction. Once no application pods are running on the node, you can retry the node removal process.
-
<nodeName>: Use the format<your-region-name>.<node-id>for naming.<your-region-name>: The name of the region where your cluster is located;<node-id>: The ECS instance ID of the node to be removed. For example, cn-hangzhou.i-i-bp1asavedmte377c3****. -
options: Optional configuration parameters, such as--force --ignore-daemonsets --delete-local-data. For additional options, use the commandkubectl drain --help.
References
-
For node removal via OpenAPI, see RemoveNodePoolNodes - Remove node pool nodes and DeleteClusterNodes - Remove cluster nodes.
-
To manage standalone nodes within a node pool, see Migrate standalone nodes to node pool.
-
ACK allows manual scale-out and scale-in of node pools by adjusting the Expected Number Of Nodes. For instructions, see Manually scale-out and scale-in node pool.
-
To add purchased ECS instances to the ACK cluster as worker nodes or re-add them to the node pool after removal, see Add existing nodes.
-
For resetting an ECS instance or purging system disk data, refer to Reinitialize the system disk (reset the operating system).