Container Service for Kubernetes:FAQs about node auto scaling

Index

What scheduling policies does cluster-autoscaler use to determine whether unschedulable pods can be scheduled to a node pool for which node auto scaling is enabled?

The following list describes the scheduling policies used by cluster-autoscaler:

• PodFitsResources

• GeneralPredicates

• PodToleratesNodeTaints

• MaxGCEPDVolumeCount

• NoDiskConflict

• CheckNodeCondition

• CheckNodeDiskPressure

• CheckNodeMemoryPressure

• CheckNodePIDPressure

• CheckVolumeBinding

• MaxAzureDiskVolumeCount

• MaxEBSVolumeCount

• ready

• NoVolumeZoneConflict

What resources can cluster-autoscaler simulate?

cluster-autoscaler can simulate and evaluate the following resources:

cpu
memory
sigma/eni
ephemeral-storage
aliyun.com/gpu-mem (shared GPUs only)
nvidia.com/gpu

For more information, see the How do I add custom resources to node pools for which auto scaling is enabled? section of this topic.

Why does cluster-autoscaler fail to add nodes after a scale-out activity is triggered?

Check whether your situation matches any of the following:

• The instance types in the scaling group may not meet the resource requests from pods. Resources provided by Elastic Compute Service (ECS) instance types comply with the ECS specifications. Some resources provided by the specified ECS instance type are reserved or occupied for the following purposes:

  • Resources are used for virtualization or occupied by the operating system during instance creation. For more information, see Why does a purchased instance have a memory size that is different from the memory size defined in the instance type?

  • ACK reserves a portion of node resources to operate essential Kubernetes components and system processes. Specifically, resources are allocated for components like kubelet, kube-proxy, Terway, and the container runtime. This reservation ensures that the OS kernel, system services, and Kubernetes daemons can run as normal. However, the amount of allocatable resources of a node is reduced compared to its total resource capacity. For more information, see Resource reservation policy.

  • By default, system components are installed on each node. Therefore, the requested pod resources must be less than the resource capacity of the instance type.

• Cross-zone scale-out activities cannot be triggered for pods that have limits on zones.

• The Resource Access Management (RAM) role may lack the necessary permissions to manage the Kubernetes cluster. You must configure RAM roles for each Kubernetes cluster that is involved in the scale-out activity. For more information, see Prerequisites.

• The following issues may occur when you enable node auto scaling:

  • The instance fails to be added to the cluster and a timeout error occurs.

  • The node is not ready and a timeout error occurs.

  To ensure accurate scaling, cluster-autoscaler will not perform any scaling activities until abnormal nodes are fixed.

How does cluster-autoscaler evaluate the resource capacity of a scaling group that uses multiple types of instances?

For a scaling group that uses multiple types of instances, cluster-autoscaler evaluates the resource capacity of the scaling group based on the least amount of resources that the scaling group can provide.

For example, a scaling group uses two types of instances. One instance type provides 4 vCores and 32 GB of memory and the other one provides 8 vCores and 16 GB of memory. In this scenario, cluster-autoscaler considers that the scaling group can add instances which each provide 4 vCores and 16 GB of memory. If a pending pod requests resources more than 4 vCores or 16 GB of memory, the pod is not scheduled.

You still need to take resource reservation into consideration after you specify multiple instance types for a scaling group. For more information, see the Why does cluster-autoscaler fail to add nodes after a scale-out activity is triggered? section of this topic.

How do I choose between multiple node pools with auto scaling enabled during elastic scaling?

When pods cannot be scheduled to nodes, the auto scaling component simulates the scheduling of the pods based on the configurations of scaling groups. The configurations include labels, taints, and instance specifications. If a scaling group meets the requirements, it is selected for the scale-out activity. If two or more scaling groups meet the requirements, node auto scaling applies the least-waste principle and selects the scaling group with the fewest idle resources after simulation.

Why does a pod fail to be scheduled to a node that is added by cluster-autoscaler?

The estimated amount of available resources on the node added by cluster-autoscaler may be more than the actual amount of resources available on the node due to the accuracy of underlying resource calculation by cluster-autoscaler. For more information, see the Why does a purchased instance have a memory size that is different from the memory size defined in the instance type? section. If the resources requested by the pods on a node exceed 70% of the computing resources provided by the node, we recommend that you check whether the pods can be scheduled to another node of the same instance type.

cluster-autoscaler checks only the resource requests of pending pods and pods created by DaemonSets when cluster-autoscaler determines whether the nodes in your cluster can provide sufficient resources for pod scheduling. If the nodes in your cluster have static pods that are not created by DaemonSets, we recommend that you reserve resources for these pods.

How do I add custom resources to node pools for which auto scaling is enabled?

To enable cluster-autoscaler to identify custom resources provided by a node pool for which auto scaling is enabled or identify the amounts of specific resource types provided by the node pool, you need to add an ECS tag with the following prefix to the node pool:

k8s.io/cluster-autoscaler/node-template/resource/{resource-name}:{resource-size}

Example:

k8s.io/cluster-autoscaler/node-template/resource/hugepages-1Gi:2Gi

Why does cluster-autoscaler fail to remove nodes after a scale-in activity is triggered?

Check whether your situation matches any of the following:

• The requested resource threshold of each pod is higher than the specified scale-in threshold.

• Pods that belong to the kube-system namespace are running on the node.

• A scheduling policy forces the pods to run on the current node. Consequently, the pods cannot be scheduled to other nodes.

• PodDisruptionBudget is set for the pods on the node and the minimum value of PodDisruptionBudget is reached.

For more information about FAQs, see autoscaler in the open-source community.

How do I enable or disable pod eviction for a specific DaemonSet pod?

cluster-autoscaler determines whether to evict DaemonSet pods based on the Evict DaemonSet Pods setting. The setting takes effect on all DaemonSet pods in the cluster. For more information, see Step 1: Enable node auto scaling. If you want to enable the DaemonSet pod eviction feature for a DaemonSet pod, add the "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "true" annotation to the pod configuration.

If you want to disable the DaemonSet pod eviction feature for a DaemonSet pod, add the "cluster-autoscaler.kubernetes.io/enable-ds-eviction": "false" annotation to the pod configuration.

What types of pods can prevent cluster-autoscaler from removing nodes?

If a pod is not created by a native Kubernetes Controller, such as a Deployment, ReplicaSet, Job, or StatefulSet, or If pods on a node cannot be securely terminated or migrated, the node may not be removed by the cluster-autoscaler. For more information, see What types of pods can prevent CA from removing a node?

Does cluster-autoscaler support CRD?

cluster-autoscaler supports only Kubernetes standard objects. It does not support Kubernetes CustomResourceDefinitions (CRDs).

How do I set a scale-out delay in cluster-autoscaler for unschedulable pods?

You can use the cluster-autoscaler.kubernetes.io/pod-scale-up-delay annotation to set a scale-out delay for each pod. If pods are still unschedulable after the delay ends, cluster-autoscaler may add nodes to schedule the pods. For example, you can use the "cluster-autoscaler.kubernetes.io/pod-scale-up-delay": "600s" annotation.

How do I use pod annotations to allow cluster-autoscaler to remove the node that hosts the pod, or prevent cluster-autoscaler from removing the pod that hosts the pod?

Pod annotations can specify whether pods prevent or allow nodes to be scaled in by cluster-autoscaler:

• To configure a pod to prevent cluster-autoscaler from removing the node that hosts the pod, add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" annotation.

• To configure a pod to allow cluster-autoscaler to remove the node that hosts the pod, add the "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" annotation to the pod configuration.

How do I prevent cluster-autoscaler from removing nodes?

To prevent cluster-autoscaler from removing nodes, add the "cluster-autoscaler.kubernetes.io/scale-down-disabled": "true" annotation to the node configurations. Run the following command to add the annotation:

kubectl annotate node <nodename> cluster-autoscaler.kubernetes.io/scale-down-disabled=true

How do I update cluster-autoscaler to the latest version?

For a cluster with auto scaling enabled, you can use one of the following methods to update cluster-autoscaler:

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.

3. Click Edit on the right side of Node Scaling, then click OK to update cluster-autoscaler to the latest version.

What operations can trigger the system to automatically update cluster-autoscaler?

To ensure that the configurations of cluster-autoscaler are up to date and its version is compatible with the cluster, the following operations can trigger the system to automatically update cluster-autoscaler:

• Update the auto scaling setting.

• Create, delete, or update node pools for which auto scaling is enabled.

• Successfully upgrade clusters.

Why does node scaling still fail after I complete role authorization in the ACK managed cluster?

This issue may be due to the absence of addon.aliyuncsmanagedautoscalerrole.token in the Secret under the kube-system namespace of the cluster. If this token is missing, use one of the following methods to add it:

• Submit a ticket for technical support.

• Manually add the permission: By default, ACK assumes the worker RAM role to use the relevant capabilities. Use the following steps to manually assign the AliyunCSManagedAutoScalerRolePolicy permission to the worker role:

  1. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, click Cluster Information.

  2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.

  3. On the Node Pools page, click Enable next to Node Scaling.

     

  4. Authorize the KubernetesWorkerRole role and the AliyunCSManagedAutoScalerRolePolicy system policy as prompted. The following figure shows the console page on which you can complete the authorization:

     

  5. To apply the new RAM policy, manually restart the cluster-autoscaler or ack-goatscaler Deployment in the kube-system namespace. The cluster-autoscaler manages node auto scaling, while ack-goatscaler handles node instant scaling.