Container Service for Kubernetes:Scheduling FAQ

How do I prevent pod startup failures due to insufficient IP addresses in a virtual switch?

How do I migrate ack-descheduler to Koordinator Descheduler?

The ack-descheduler add-on is no longer maintained. For more information, see [Notice] ack-descheduler migration. We recommend that you migrate to the Koordinator Descheduler add-on, which is actively maintained. The migration process is similar to migrating Kubernetes Descheduler to Koordinator Descheduler. For more information, see Migrate Kubernetes Descheduler to Koordinator Descheduler.

Container Service for Kubernetes (ACK) provides the ack-descheduler add-on in the marketplace. This add-on is based on the open source Kubernetes Descheduler. Versions 0.20.0 and 0.27.1 of ack-descheduler are available. They provide the same features and are used in the same way as the corresponding open source Kubernetes Descheduler versions.

What is the default scheduling policy of ACK Scheduler?

In an ACK cluster, the default scheduling policy of ACK Scheduler is consistent with the community Kubernetes scheduler. When the Kubernetes scheduler decides how to schedule a pod to a node, the process typically involves two key steps: Filter and Score.

• Filter: Filters the nodes where the pod can be scheduled. If the filtered list is empty, the pod cannot be scheduled.

• Score: After filtering, the scheduler scores and ranks the schedulable nodes to select the most suitable node for the pod.

For more information about the Filter and Score plug-ins that are enabled in the latest version of ACK Scheduler, see kube-scheduler.

How do I avoid nodes with high resource utilization during pod scheduling?

In the native Kubernetes scheduling policy, the scheduler makes decisions based on the resource requests of nodes, not their actual resource utilization. Additionally, various Filter and Score plug-ins affect the scheduling result. You can use the following features provided by ACK clusters to prevent pods from being scheduled to nodes with high resource utilization.

• Configure appropriate resource requests and limits for each pod and plan for resource redundancy. You can use the resource profiling feature to obtain recommended container specifications based on historical resource usage data. For more information, see Resource profiling.

• Enable the load-aware pod scheduling feature. The load-aware pod scheduling feature of the kube-scheduler add-on provided by ACK is based on the Kubernetes scheduling framework. Unlike native Kubernetes scheduling policies, ACK Scheduler can perceive the actual resource usage of nodes. It analyzes historical node load data, estimates the resource requirements for new pods, and then preferentially schedules the pods to nodes with lower loads to achieve load balancing. For more information, see Load-aware scheduling.

• Enable load-aware hotspot descheduling. Node utilization changes dynamically over time due to variations in the cluster environment, workload traffic, or requests. This can disrupt the load balance among nodes in the cluster. To address this, ACK Scheduler provides a descheduling capability to prevent extreme load imbalances. For more information, see Using load-aware hotspot descheduling.

A new node is added to the cluster. Why are pods not scheduled to it?

This issue can occur for several reasons. You can troubleshoot the issue as follows.

1. Check whether the node status is normal. If the node is in the `NotReady` state, it cannot accept pods.

2. Check whether the pod has scheduling policies, such as `NodeSelector`, `NodeAffinity`, or `PodAffinity`, or has taints that prevent it from being scheduled to the new node.

3. Evaluate whether the issue is related to the native Kubernetes scheduling policy. In the native Kubernetes scheduling policy, the scheduler makes decisions based on the resource requests of nodes, not their actual resource utilization. This can result in some nodes running many pods while others run few or no pods.

   For more information about the solutions to this issue, see How do I avoid nodes with high resource utilization during pod scheduling?

Why does scheduling fail due to insufficient CPU or memory when the cluster's overall utilization is not high?

In the native Kubernetes scheduling policy, the scheduler makes decisions based on the resource requests of nodes, not their actual resource utilization. Therefore, even if the actual CPU utilization of the cluster is not high, pod scheduling may fail due to insufficient CPU or memory.

For more information about the solutions to this issue, see How do I avoid nodes with high resource utilization during pod scheduling?

What do I need to know when I use the descheduling feature in ACK? Will it restart pods?

ACK provides the descheduling feature through Koordinator Descheduler. When you use the descheduling feature, note the following.

• Koordinator Descheduler is responsible only for evicting running pods, not for their recreation or subsequent scheduling. The recreation of a pod after eviction is handled by its corresponding workload controller, such as a Deployment or StatefulSet. The scheduling of the recreated pod is handled by the scheduler.

• During descheduling, the old pod is evicted before the new pod is created. Ensure that your application has a sufficient number of replicas (replicas) to avoid affecting application availability during eviction.

For more information, see Descheduling.

How do I schedule an application to a specific node?

You can add labels to nodes and then add a corresponding nodeSelector to your application's YAML file to schedule the application to a specific node. For more information, see Schedule an application to a specific node.

In a deployment, how can I schedule a specific number of pods to ECS and a specific number to ECI?

In scenarios where ECS and ECI resources are used together, you can use UnitedDeployment to define subsets for workload management. For example, in the YAML file of a UnitedDeployment, you can set replicas to 10 in subset-ecs and replicas to 10 in subset-eci. For more information, see Scale workloads based on UnitedDeployment.

How can I ensure high availability for a workload's pods during scheduling?

You can use pod affinity to distribute a workload's pods across different zones or nodes. For example, you can add the following fields to the pod's YAML file to declare a preferredDuringSchedulingIgnoredDuringExecution rule. This rule attempts to schedule pods with the security=S2 label to different zones. If this condition cannot be met, the scheduler attempts to schedule the pod to other nodes.

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S2
          topologyKey: topology.kubernetes.io/zone

For more information, see podAffinity and podAntiAffinity and Pod Topology Spread Constraints in the Kubernetes documentation.

For a batch of newly created pods, why are they not all scheduled to the node with the lowest load?

If the scheduler schedules a batch of newly created pods to the node with the lowest current load, that node might quickly become a hot spot because the new pods will increase the node's load after they start.

Therefore, when the load-aware scheduling plug-in calculates node scores, it adjusts the score if a node has new pods whose utilization has not yet been reported. This prevents over-scheduling and the creation of new hot spots.

Besides the node load level, what other factors affect the scheduling result?

The Kubernetes scheduler consists of multiple plug-ins. During scheduling, many plug-ins participate in scoring and ranking nodes, such as the affinity plug-in and the topology spread plug-in. The final ranking of nodes is influenced by all these plug-ins. You can adjust the scoring weights of each plug-in as needed.

After the scheduler is upgraded to a new version, is the load-aware scheduling feature that uses the old protocol still supported?

To use the old version of the load-aware scheduling feature, you must add the annotation alibabacloud.com/loadAwareScheduleEnabled: "true" to the pod.

The following tables describe the protocol support and add-on version requirements.

After I enable the CPU Burst configuration, why are my pods still throttled?

This issue can occur for several reasons. Refer to the following solutions to make adjustments.

• The configuration format is incorrect, which prevents the CPU Burst policy from taking effect. For more information, see Advanced parameter configuration to modify and verify the configuration.

• If the CPU utilization reaches the upper limit specified by cfsQuotaBurstPercent, CPU throttling still occurs due to insufficient CPU resources.

  Adjust the request and limit values based on your application's actual requirements.

• The CPU Burst policy adjusts two cgroup parameters for a pod: cpu.cfs_quota_us and cpu.cfs_burst_us. For more information, see Advanced Parameter Configuration. The cpu.cfs_quota_us parameter is set only after ack-koordinator detects CPU throttling, which causes a slight latency. In contrast, the cpu.cfs_burst_us parameter is set directly based on the configuration and is more responsive.

  For better performance, use this feature with an Alibaba Cloud Linux operating system.

• The CPU Burst policy includes a protection mechanism for adjusting cpu.cfs_quota_us, which uses the sharePoolThresholdPercent parameter to define the machine's safety threshold. If the overall machine utilization becomes too high, cpu.cfs_quota_us is reset to its initial value to prevent a single pod from causing excessive interference.

  You should set the node's safety threshold based on your application's requirements to prevent high node utilization from affecting application performance.

Is Alibaba Cloud Linux required to enable the CPU Burst policy?

The ack-koordinator CPU Burst policy is compatible with all Alibaba Cloud Linux and CentOS open source kernels. However, we recommend that you use Alibaba Cloud Linux. By leveraging the features of the Alibaba Cloud Linux kernel, ack-koordinator can provide a more fine-grained CPU elasticity management mechanism. For more information, see Enable the CPU Burst feature on the cgroup v1 interface.

After an application uses Batch resources, why does its memory usage suddenly increase?

For applications configured with a Batch memory limit (kubernetes.io/batch-memory), ack-koordinator sets the container's cgroup memory limit after the container is created. Some applications request memory based on the cgroup limit at startup. If an application starts before ack-koordinator configures the cgroup limit, its actual memory usage might exceed the intended Batch memory limit. The operating system does not immediately reduce the process's memory usage. As a result, the cgroup memory limit cannot be applied until the actual usage falls below the Batch memory limit.

In this case, you can adjust the application's configuration parameters to control its memory usage so that it stays below the Batch limit. Alternatively, you can check the memory limit parameter in the application's startup script and ensure it is set before the application starts. This helps ensure that the application's memory usage is reasonably limited and prevents issues such as out-of-memory (OOM) errors.

You can run the following command in the container to view the memory resource limit parameter.

# The unit is bytes.
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
# Expected output.
1048576000

After the number of CPU cores is increased, why does performance decrease and pods experience CPU throttling issues?

Will the dynamic resource overselling feature of the legacy ack-slo-manager protocol be supported after an upgrade to the ack-koordinator add-on?

The old version of the dynamic resource overcommitment protocol includes two parts:

• The alibabacloud.com/qosClass annotation in the pod.

• The alibabacloud.com/reclaimed resource in the pod's requests and limits.

ack-koordinator is compatible with the preceding older protocols and uniformly calculates the resource requests and available resources for both new and old protocols in the scheduler of an ACK Pro managed cluster. You can seamlessly upgrade your add-on to ack-koordinator.

The scheduler in ACK Pro managed clusters and ack-koordinator support different protocol versions as follows.

If the CPU Burst feature is used with the old protocol of ack-slo-manager, is it still supported after an upgrade to ack-koordinator?

The previous version of the Pod protocol requires you to specify alibabacloud.com/cpuBurst in the Annotation. ack-koordinator is fully compatible with this previous version, allowing you to seamlessly upgrade the add-on to a new version.

The compatibility of ack-koordinator with different protocol versions is as follows.

I am using the CPU QoS feature with the old protocol of ack-slo-manager. Is this feature still supported after I upgrade to ack-koordinator?

In the old pod protocol (version 0.8.0 and earlier), CPU QoS is enabled by adding the alibabacloud.com/qosClass annotation to the pod.

ack-koordinator remains compatible with older protocols. You can seamlessly upgrade your add-on to ack-koordinator and gradually switch your Pods to the koordinator.sh protocol. ack-koordinator will remain compatible with these older protocols until July 30, 2023. We recommend that you promptly upgrade the resource fields of the original protocol to the new version.

Is the container memory QoS feature still supported after upgrading from the ack-slo-manager legacy protocol to ack-koordinator?

The old pod protocol (version 0.8.0 and earlier) includes:

• The alibabacloud.com/qosClass annotation in the pod.

• The alibabacloud.com/memoryQOS annotation in the pod.

ack-koordinator is compatible with previous protocol versions, allowing you to seamlessly upgrade the add-on to ack-koordinator. Backward compatibility is supported until July 30, 2023. We recommend that you promptly upgrade the resource fields of the original protocol to the new version.

ack-koordinator supports the memory QoS feature in different versions as follows.

The node utilization has reached the threshold, but pods on the node are not evicted. What should I do?

This issue can occur for several reasons. Refer to the following solutions to make adjustments.

Why does the descheduler restart frequently?

This can happen if the descheduler's ConfigMap format is incorrect or if the ConfigMap does not exist. Check the content and format of the ConfigMap file and make the necessary corrections. For more information, see Advanced configuration parameters. After modifying the ConfigMap, you must restart the descheduler. For more information about how to restart the descheduler, see Step 2: Enable the descheduling plug-in to distribute pods on nodes with high resource utilization.

How do I use load-aware scheduling and hot spot descheduling together?

After you enable descheduling for nodes with high resource utilization, pods on those nodes are evicted. The scheduler then schedules the new pods, which are created by the parent controller such as a Deployment, to suitable nodes based on its configuration. To achieve better load balancing, you should also enable load-aware scheduling for the scheduler. For more information, see Use load-aware scheduling.

When you configure these features, set the loadAwareThreshold parameter for the node filtering function in load-aware scheduling to the same value as the highThresholds parameter in the descheduler. For more information, see Policy description. When a node's load exceeds highThresholds, the descheduler evicts pods from that node. The scheduler then uses loadAwareThreshold to prevent new pods from being scheduled back to that node. Otherwise, evicted pods might be rescheduled to their original node, especially if the pod has a specified node scheduling scope with a small number of nodes that have similar utilization levels.

What is the utilization algorithm used for descheduling?

The descheduler continuously monitors resource utilization over a period and calculates the average value. Descheduling is triggered only if the average value remains above the threshold for a specific duration. The default duration is approximately 10 minutes. For memory resources, the descheduler's memory usage calculation excludes the page cache because this memory can be reclaimed by the operating system. However, the utilization value returned by kubectl top node includes the page cache. You can use Monitoring with Alibaba Cloud Prometheus to view the actual memory usage.

During a stress test with wrk, the result shows "Socket errors: connect 54,". What should I do?

Problem Description

During a stress test with wrk, the result shows Socket errors: connect 54,. The wrk client fails to establish a connection with the Nginx server, and the test result is invalid.

Cause

This error usually occurs because the number of available connections on the client is limited, which causes connection attempts to fail.

Solution

To resolve this error, check the TCP connection configuration on the stress testing machine and enable TCP connection reuse.

1. Log on to the stress testing machine and run the following command to check whether TCP connection reuse is enabled.

   sudo sysctl -n net.ipv4.tcp_tw_reuse

   An output of 0 or 2 indicates that the system has not fully enabled TCP connection reuse.

2. Run the following command to enable TCP connection reuse.

   sudo sysctl -w net.ipv4.tcp_tw_reuse=1

3. Use the wrk tool to send stress testing requests again.

   Verify that the test results no longer include the Socket errors: connect 54, ... error message, which indicates that the results are valid.

Why is no data displayed in the cluster colocation benefits section on the k8s-reclaimed-resource tab?

1. Check whether the ack-koordinator add-on is installed.

   1. Log on to the ACK console. In the navigation pane on the left, click Clusters.

   2. On the Clusters page, find the cluster you want and click its name. In the left navigation pane, choose Applications > Helm.

   3. On the Helm page, check whether the ack-koordinator add-on exists.

      • If it does not exist, see Install and manage add-ons, install the ack-koordinator add-on, and then follow the steps below.

      • If it exists, proceed to the next step.

2. Check whether the colocation of multi-types workloads dashboard displays relevant data.

   If not, perform the following steps:

   1. Log on to the ARMS console.

   2. In the left navigation pane, choose Managed Service for Prometheus > Instances.

   3. In the upper-left corner of the page, select the destination region, click the name of the Prometheus instance, and then click Metric Management in the navigation pane on the left.

   4. Click the blue Metrics button, search for and select kube_node_labels as prompted, and view the metric's data details.

Can I use Arm-based spot instances?

Yes, Arm-based spot instances are available. For more information about how to use them, see Use spot instances.

What are the limitations of using Arm-based nodes in an ACK cluster?