Container Service for Kubernetes:Work with load-aware hotspot descheduling

Limits

• Only ACK Pro clusters support load-aware hotspot descheduling. For more information, see Create an ACK managed cluster.

• To use the load-aware hotspot descheduling feature, make sure that the following requirements are met.

  Component	Required version
  ACK scheduler	v1.22.15-ack-4.0 and later or v1.24.6-ack-4.0 and later
  ack-koordinator	v1.1.1-ack.1 and later
  Helm	3.0 and later

Billing

No fee is charged when you install or use the ack-koordinator component. However, fees may be charged in the following scenarios:

• ack-koordinator is a non-managed component that occupies worker node resources after it is installed. You can specify the amount of resources requested by each module when you install the component.

• By default, ack-koordinator exposes the monitoring metrics of features such as resource profiling and fine-grained scheduling as Prometheus metrics. If you enable Prometheus metrics for ack-koordinator and use Managed Service for Prometheus, these metrics are considered custom metrics and fees are charged for these metrics. The fee depends on factors such as the size of your cluster and the number of applications. Before you enable Prometheus metrics, we recommend that you read the Billing topic of Managed Service for Prometheus to learn about the free quota and billing rules of custom metrics. For more information about how to monitor and manage resource usage, see Query the amount of observable data and bills.

Introduction to load-aware hotspot descheduling

Load-aware pod scheduling

The ACK scheduler supports load-aware scheduling, which can schedule pods to nodes that run with low loads. Due to the changes in the cluster environment, traffic, and requests, the utilization of nodes dynamically changes and may break the load balance between nodes in the cluster, and even result in extreme load imbalance. This affects the runtime quality of the workload. ack-koordinator can identify changes in the loads of nodes and automatically optimize nodes that exceed the safety load to prevent extreme load imbalance. You can use a combination of load-aware scheduling and hotspot descheduling to achieve optimal load balancing among nodes. For more information, see Use load-aware pod scheduling.

How Koordinator Descheduler works

Koordinator Descheduler is a module of the ack-koordinator component. The LowNodeLoad plug-in can identify the changes in loads and perform hotspot descheduling. Unlike the Kubernetes-native descheduler plug-in LowNodeUtilization, the LowNodeLoad plug-in decides to deschedule nodes based on the actual utilization of the nodes, while LowNodeUtilization decides to deschedule based on resource allocation.

Descheduling procedure

Koordinator Descheduler periodically performs descheduling. The following figure shows the steps of descheduling within each cycle.

1. Data collection: collects information about nodes and workloads in the cluster and the resource utilization statistics.

2. Descheduling policy implementation by the policy plug-in.

   This section uses LowNodeLoad as an example.

   1. Identifies hotspot nodes. For more information about the classification of nodes, see Load thresholds.

   2. Traverses all hotspot nodes, identifies the nodes on which pods can be migrated, and sorts the pods. For more information about how pods are scored and sorted, see Pod scoring policy.

   3. Traverses all pods to be migrated and checks whether the pods meet the requirements for migration based on constraints such as the cluster size, resource utilization, and the ratio of replicated pods. For more information, see Load-aware hotspot descheduling policies.

   4. Only pods that meet the requirements are migrated. If no pod meets the requirements on the current node, LowNodeLoad continues to traverse the pods on other hotspot nodes.

3. Pod eviction and migration: evicts the pods that meet the requirements for migration. For more information, see API-initiated Eviction.

Load thresholds

In the following figure, lowThresholds is set to 45% and highThresholds is set to 70%. Nodes are classified based on their loads and the thresholds. If the values of lowThresholds and highThresholds change, the standards for node classification also change.

The resource utilization statistics are updated every minute and the average values within the previous 5 minutes are displayed.

• Idle nodes: nodes whose resource utilization is lower than 45%.

• Normal nodes: nodes whose resource utilization is higher than or equal to 45% but lower than or equal to 70%. This is the desired resource utilization range for cluster nodes.

• Hotspot nodes: nodes whose resource utilization is higher than 70. Pods on hotspot nodes will be evicted until the resource utilization of these nodes drops to 70% or lower.

Load-aware hotspot descheduling policies

kubectl get event | grep stress-demo-588f9646cf-7****
55s         Normal    Evicting           podmigrationjob/3bf8f623-4d10-4fc5-ab4e-2bead3c4****   Pod "default/stress-demo-588f9646cf-7****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(76.72%)>threshold(50.00%)"
22s         Normal    EvictComplete      podmigrationjob/3bf8f623-4d10-4fc5-ab4e-2bead3c4****   Pod "default/stress-demo-588f9646cf-7****" has been evicted
55s         Normal    Descheduled        pod/stress-demo-588f9646cf-7****                       Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(76.72%)>threshold(50.00%)"
55s         Normal    Killing            pod/stress-demo-588f9646cf-7****                       Stopping container stress

Step 1: Install or modify ack-koordinator and enable load-aware hotspot descheduling

Step 2: Enable the LowNodeLoad plug-in

1. Create a file named koord-descheduler-config.yaml and add the following YAML content to the file:

   The koord-descheduler-config.yaml file is a ConfigMap used to enable the LowNodeLoad plug-in.

   Click to view details

   # koord-descheduler-config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: koord-descheduler-config
     namespace: kube-system
   data:
     koord-descheduler-config: |
       # Do not modify the following system configuration of koord-desheduler. 
       apiVersion: descheduler/v1alpha2
       kind: DeschedulerConfiguration
       leaderElection:
         resourceLock: leases
         resourceName: koord-descheduler
         resourceNamespace: kube-system
       deschedulingInterval: 120s # The interval at which LowNodeLoad runs. The interval is set to 120 seconds in this example. 
       dryRun: false # The global read-only mode. After you enable this mode, Koordinator Descheduler does not perform any operations. 
       # The preceding configuration is the system configuration. 
   
       profiles:
       - name: koord-descheduler
         plugins:
           deschedule:
             disabled:
               - name: "*"
           balance:
             enabled:
               - name: LowNodeLoad # Enable the LowNodeLoad plug-in. 
           evict:
             disabled:
               - name: "*"
             enabled:
               - name: MigrationController # Enable the migration controller. 
   
         pluginConfig:
         - name: MigrationController # Configure the parameters of the migration controller. 
           args:
             apiVersion: descheduler/v1alpha2
             kind: MigrationControllerArgs
             defaultJobMode: EvictDirectly
   
         - name: LowNodeLoad # Configure the LowNodeLoad plug-in. 
           args:
             apiVersion: descheduler/v1alpha2
             kind: LowNodeLoadArgs
   
             lowThresholds:  # Specify the low resource threshold for identifying idle nodes. Nodes whose load is lower than the threshold are considered idle nodes. 
               cpu: 20 # The low CPU utilization threshold is 20%. 
               memory: 30  # The low memory utilization threshold is 30%. 
             highThresholds: # Specify the high resource threshold for identifying hotspot nodes. Nodes whose load is higher than the threshold are considered hotspot nodes. 
               cpu: 50  # The high CPU utilization threshold is 50%. 
               memory: 60 # The high memory utilization threshold is 60%. 
   
             evictableNamespaces: # Specify the namespaces that you want to include or exclude for descheduling. The include and exclude lists are mutually exclusive. 
               include: # Specify the namespaces that you want to include for descheduling. 
                 - default
               # exclude: # Specify the namespaces that you want to exclude for descheduling. 
                 # - "kube-system"
                 # - "koordinator-system"

2. Run the following command to apply the configuration to the cluster:

   kubectl apply -f koord-descheduler-config.yaml

3. Run the following command to restart Koordinator Descheduler.

   After Koordinator Descheduler is restarted, the modified configuration takes effect.

   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 0
   deployment.apps/ack-koord-descheduler scaled
   kubectl -n kube-system scale deploy ack-koord-descheduler --replicas 1
   deployment.apps/ack-koord-descheduler scaled

Step 3: (Optional) Enable the load-aware scheduling plug-in

Enable the load-aware scheduling plug-in to achieve optimal load balancing among nodes. For more information, see Step 1: Enable load-aware scheduling.

Step 4: Verify load-aware hotspot descheduling

In this section, a cluster that contains three nodes is used as an example. Each node has 104 vCores and 396 GB of memory.

1. Create a file named stress-demo.yaml and add the following content to the file.

   Click to view details

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: stress-demo
     namespace: default
     labels:
       app: stress-demo
   spec:
     replicas: 6
     selector:
       matchLabels:
         app: stress-demo
     template:
       metadata:
         name: stress-demo
         labels:
           app: stress-demo
       spec:
         containers:
           - args:
               - '--vm'
               - '2'
               - '--vm-bytes'
               - '1600M'
               - '-c'
               - '2'
               - '--vm-hang'
               - '2'
             command:
               - stress
             image: polinux/stress
             imagePullPolicy: Always
             name: stress
             resources:
               limits:
                 cpu: '2'
                 memory: 4Gi
               requests:
                 cpu: '2'
                 memory: 4Gi
         restartPolicy: Always

2. Run the following command to create a pod for stress testing:

   kubectl create -f stress-demo.yaml
   deployment.apps/stress-demo created

3. Run the following command to view the status of the pod until it starts to run:

   kubectl get pod -o wide

   Expected output:

   NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE                    NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Running   0          82s   10.XX.XX.53   cn-beijing.10.XX.XX.53   <none>           <none>

   The output indicates that pod stress-demo-588f9646cf-s**** is scheduled to node cn-beijing.10.XX.XX.53.

4. Increase the load of node cn-beijing.10.XX.XX.53. Then, run the following command to check the load of each node:

   kubectl top node

   Expected output:

   NAME                      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
   cn-beijing.10.XX.XX.215   17611m       17%    24358Mi         6%
   cn-beijing.10.XX.XX.53    63472m       63%    11969Mi         3%

   The output indicates that the load of node cn-beijing.10.XX.XX.53 is higher, which is 63%. The load exceeds the high resource threshold of 50%. The load of node cn-beijing.10.XX.XX.215 is lower, which is 17%. The load is lower than the low resource threshold of 20%.

5. Enable load-aware hotspot descheduling. For more information, see Step 2: Enable the LowNodeLoad plug-in.

6. Run the following command to view the changes of the pods.

   Wait for the descheduler to identify hotspot nodes and evict pods.

   Note

   A node is considered a hotspot node if the node consecutively exceeds the high resource threshold 5 times within 10 minutes.

   kubectl get pod -w

   Expected output:

   NAME                           READY   STATUS               RESTARTS   AGE     IP           NODE                     NOMINATED NODE   READINESS GATES
   stress-demo-588f9646cf-s****   1/1     Terminating          0          59s   10.XX.XX.53    cn-beijing.10.XX.XX.53     <none>           <none>
   stress-demo-588f9646cf-7****   1/1     ContainerCreating    0          10s   10.XX.XX.215   cn-beijing.10.XX.XX.215    <none>           <none>

7. Run the following command to view the event:

   kubectl get event | grep stress-demo-588f9646cf-s****

   Expected output:

   2m14s       Normal    Evicting            podmigrationjob/00fe88bd-8d4c-428d-b2a8-d15bcdeb****   Pod "default/stress-demo-588f9646cf-s****" evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   101s        Normal    EvictComplete       podmigrationjob/00fe88bd-8d4c-428d-b2a8-d15bcdeb****   Pod "default/stress-demo-588f9646cf-s****" has been evicted
   2m14s       Normal    Descheduled         pod/stress-demo-588f9646cf-s****                       Pod evicted from node "cn-beijing.10.XX.XX.53" by the reason "node is overutilized, cpu usage(68.53%)>threshold(50.00%)"
   2m14s       Normal    Killing             pod/stress-demo-588f9646cf-s****                       Stopping container stress

   The output indicates the migration result. The pods on the hotspot node are migrated to the idle node.

Advanced settings

Advanced settings for Koordinator Descheduler

The configuration of Koordinator Descheduler is stored in a ConfigMap. The following code block shows the advanced settings for load-aware hotspot descheduling.

# koord-descheduler-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: koord-descheduler-config
  namespace: kube-system
data:
  koord-descheduler-config: |
    # Do not modify the following system configuration of koord-desheduler. 
    apiVersion: descheduler/v1alpha2
    kind: DeschedulerConfiguration
    leaderElection:
      resourceLock: leases
      resourceName: koord-descheduler
      resourceNamespace: kube-system
    deschedulingInterval: 120s # The interval at which LowNodeLoad runs. The interval is set to 120 seconds in this example. 
    dryRun: false # The global read-only mode. After you enable this mode, koord-descheduler does not perform any operations. 
    # The preceding configuration is the system configuration. 

    profiles:
    - name: koord-descheduler
      plugins:
        deschedule:
          disabled:
            - name: "*"
        balance:
          enabled:
            - name: LowNodeLoad # Enable the LowNodeLoad plug-in. 
        evict:
          disabled:
            - name: "*"
          enabled:
            - name: MigrationController # Enable the migration controller. 

      pluginConfig:
      - name: MigrationController # Configure the parameters of the migration controller. 
        args:
          apiVersion: descheduler/v1alpha2
          kind: MigrationControllerArgs
          defaultJobMode: EvictDirectly
          maxMigratingPerNode: 1 # The maximum number of pods that can be migrated at the same time on a node. 
          maxMigratingPerNamespace: 1  # The maximum number of pods that can be migrated at the same time in a namespace. 
          maxMigratingPerWorkload: 1 # The maximum number of pods that can be migrated at the same time in a workload, such as a Deployment. 
          maxUnavailablePerWorkload: 2 # The maximum number of unavailable replicated pods that are allowed in a workload, such as a Deployment. 
          evictLocalStoragePods: false # Specify whether pods that are configured with the emptyDir or hostPath can be descheduled.
          objectLimiters:
            workload: # Control workload-specific pod migration. By default, the system can migrate only one replicated pod within 5 minutes after the first eviction. 
              duration: 5m
              maxMigrating: 1

      - name: LowNodeLoad # Configure the LowNodeLoad plug-in. 
        args:
          apiVersion: descheduler/v1alpha2
          kind: LowNodeLoadArgs

          lowThresholds:  # Specify the low resource threshold for identifying idle nodes. Nodes whose load is lower than the threshold are considered idle nodes. 
            cpu: 20 # The low CPU utilization threshold is 20%. 
            memory: 30  # The low memory utilization threshold is 30%. 
          highThresholds: # Specify the high resource threshold for identifying hotspot nodes. Nodes whose load is higher than the threshold are considered hotspot nodes. 
            cpu: 50  # The high CPU utilization threshold is 50%. 
            memory: 60 # The high memory utilization threshold is 60%. 

          anomalyCondition: # The hotspot node detection setting. 
            consecutiveAbnormalities: 5 # A node is considered a hotspot node if the node exceeds the highThresholds within five consecutive hotspot detection cycles. Hotspot nodes are descheduled and then the counter is reset. 

          evictableNamespaces: # Specify the namespaces that you want to include or exclude for descheduling. The include and exclude lists are mutually exclusive. 
            include: # Specify the namespaces that you want to include for descheduling. 
              - default
            # exclude: # Specify the namespaces that you want to exclude for descheduling. 
              # - "kube-system"
              # - "koordinator-system"

          nodeSelector: # Only the specified nodes can be descheduled. 
            matchLabels:
              alibabacloud.com/nodepool-id: np77f520e1108f47559e63809713ce****

          podSelectors: # Only the specified pods can be descheduled. 
          - name: lsPods
            selector:
              matchLabels:
                koordinator.sh/qosClass: "LS"

Koordinator Descheduler system settings

Migration control settings

type MigrationObjectLimiter struct {
    Duration time.Duration `json:"duration,omitempty"`
    MaxMigrating *intstr.IntOrString `json:"maxMigrating,omitempty"`
}

objectLimiters:
  workload:
    duration: 5m
    maxMigrating: 1

LowNodeLoad settings

highThresholds:
  cpu: 55 # The high CPU utilization threshold is set to 55%. 
  memory: 75 # The high memory utilization threshold is set to 75%.

lowThresholds:
  cpu: 25 # The low CPU utilization threshold is set to 25%. 
  memory: 25 # The low memory utilization threshold is set to 25%. 

evictableNamespaces:
  exclude:
    - "kube-system"
    - "koordinator-system"

type PodSelector struct {
    name     string
    selector metav1.LabelSelector
}

# Only Latency-Sensitive (LS) pods can be descheduled. 
podSelectors:
- name: lsPods
  selector:
    matchLabels:
      koordinator.sh/qosClass: "LS"

FAQ

What do I do if the resource utilization of a node has reached the high threshold but no pod on the node is evicted?

The following table describes the possible causes.

Why does the descheduler frequently restart?

The format of the ConfigMap of the descheduler is invalid or the ConfigMap does not exist. Refer to Advanced settings and check the content and format of the ConfigMap, modify the ConfigMap, and then restart the descheduler. For more information about how to restart the descheduler, see Step 2: Enable the LowNodeLoad plug-in.

How do I use a combination of load-aware scheduling and load-aware hotspot descheduling?

After you enable load-aware hotspot descheduling, pods on hotspot nodes are evicted. The ACK scheduler will select proper nodes for pods that are created by controllers (such as Deployments) in the upper layer. To achieve optimal load balancing, we recommend that enable load-aware scheduling at the same time. For more information, see Use load-aware scheduling.

We recommend that you set the loadAwareThreshold parameter of the scheduler and the highThresholds parameter of the descheduler to the same value. For more information, see Scheduling policies. When the load of a node exceeds highThresholds, the descheduler evicts pods on the node. The scheduler stops scheduling new pods to the hotspot node due to loadAwareThreshold. If you do not set the parameters to the same value, pods may be scheduled to the hotspot node. This issue is more likely to occur when a pod has specified the scope of schedulable nodes but only a small number of nodes are available and the resource utilization values of these nodes are close.

What is the utilization algorithm used by the descheduler?

The descheduler continuously monitors the resource utilization within a period of time and calculates the average value. Descheduling is triggered only if the average value remains above the threshold for a certain period of time. The default time period is 10 minutes. In addition, the memory utilization counted by the descheduler excludes the page cache because the memory resource used by the page cache can be recycled by the operating system. The memory utilization queried by using the kubectl top node command includes the page cache. You can view the actual memory utilization in the Managed Service for Prometheus console.