Container Service for Kubernetes:Enable dynamic resource overcommitment

1. Enable dynamic resource overcommitment

You can use the ConfigMap to enable the dynamic resource overcommitment feature. You can also configure related parameters in the ConfigMap to flexibly manage reclaimed resources, such as the reclaim threshold of resources and the policy for calculating the resource capacity of the node.

1. Create a file named configmap.yaml based on the following ConfigMap content:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ack-slo-config
     namespace: kube-system
   data:
     colocation-config: |
       {
         "enable": true,
         "metricAggregateDurationSeconds": 60,
         "cpuReclaimThresholdPercent": 60,
         "memoryReclaimThresholdPercent": 70,
         "memoryCalculatePolicy": "usage"
       }

   You can flexibly manage the batch-cpu and batch-memory resources by modifying the parameters in the ConfigMap.

   Parameters

   Parameter	Format	Description
   enable	Boolean	Specifies whether to dynamically update the statistics about Batch resources. If you disable this feature, the amount of reclaimed resources is reset to 0. Default value: false.
   metricAggregateDurationSeconds	Int	The minimum frequency at which the statistics about Batch resources are updated. Unit: seconds. We recommend that you use the default value, which is 60 seconds.
   cpuReclaimThresholdPercent	Int	The reclaim threshold of batch-cpu resources. Default value: 65. Unit: percentage (%).
   memoryReclaimThresholdPercent	Int	The reclaim threshold of batch-memory resources. Default value: 65. Unit: percentage (%).
   memoryCalculatePolicy	String	The policy for calculating the amount of batch-memory resources. Valid values: "usage": The amount of batch-memory resources is calculated based on the actual memory usage of pods whose QoS classes are Burstable or Guaranteed. The batch-memory resources include resources that are not allocated and resources that are allocated but are not in use. This is the default value. "request": The amount of batch-memory resources is calculated based on the memory requests of pods whose QoS classes are Burstable or Guaranteed. The batch-memory resources include only resources that are not allocated.

   Calculate the amount of Batch resources

   Calculate the amount of Batch resources based on the actual resource usage

   The amount of Batch resources on a node is calculated based on the actual resource usage, which includes resources that are not allocated and resources that are allocated but are not in use. You can use the following formula to calculate the amount of batch-cpu resources and the amount of batch-memory resources:

   nodeBatchAllocatable = nodeAllocatable * thresholdPercent - podUsage(non-BE) - systemUsage

   • nodeAllocatable: the amount of CPU or memory resources that can be allocated on the node.

   • thresholdPercent: the threshold of resources in percentage. You can modify this parameter as described in the preceding table.

   • podUsage(non-BE): the resource usage of pods whose QoS classes are Burstable or Guaranteed.

   • systemUsage: the usage of system resources on the node.

   Calculate the amount of Batch resources based on the resource requests of pods

   The batch-memory resources can be calculated based on the resource requests of pods. The batch-memory resources include only resources that are not allocated. For more information, see the memoryCalculatePolicy parameter in Parameters.

   You can calculate the amount of batch-memory resources based on the following formula:

   nodeBatchAllocatable = nodeAllocatable * thresholdPercent - podRequest(non-BE) - systemUsage

   podRequest(non-BE) refers to the sum of resource requests of pods whose QoS classes are Burstable and Guaranteed.

2. Check whether the ConfigMap named ack-slo-config exists in the kube-system namespace.

   • If the ack-slo-config ConfigMap exists, we recommend that you run the kubectl patch command to update the ConfigMap. This avoids changing other settings in the ConfigMap.

     kubectl patch cm -n kube-system ack-slo-config --patch "$(cat configmap.yaml)"

   • If the ack-slo-config ConfigMap does not exist, run the following command to create a ConfigMap:

     kubectl apply -f configmap.yaml

2. Apply for Batch resources for pods

After the configuration is complete, you can specify the QoS class of the pod by using the koordinator.sh/qosClass label in the metadata field of the YAML file of the pod based on the total Batch resources of the node, and specify the Batch resource request and Batch resource limit.

If you do not add the koordinator.sh/qosClass label to the pod, ack-koordinator selects a Kubernetes-native QoS class. BE indicates BestEffort pods and LS indicates Burstable or Guaranteed pods.

1. Run the following command to query the total amount of available Batch resources on the node:

   # Replace $nodeName with the name of the node that you want to query. 
   kubectl get node $nodeName -o yaml

   Expected output:

   # Pod information. 
   status:
     allocatable:
       # Unit: millicores. In the following example, 50 cores can be allocated. 
       kubernetes.io/batch-cpu: 50000
       # Unit: bytes. In the following example, 50 GB of memory can be allocated. 
       kubernetes.io/batch-memory: 53687091200

2. Create a pod and apply for Batch resources.

   Important

   • If you provision a pod by using a Deployment or other types of workloads, configure the template.metadata field. A pod cannot apply for Batch resources and regular resources at the same time.

   • ack-koordinator dynamically adjusts the Batch resources that can be allocated to pods based on the actual loads of the node. In rare cases, the kubelet may have a certain delay in synchronizing node status information. As a result, pods fail to be scheduled due to insufficient resources. In this case, delete and recreate the pods.

   • You must set the amount of extended resources to an integer in Kubernetes clusters. The unit of batch-cpu resources is millicores.

   metadata:
     labels:
       # Required. Set the QoS class of the pod to BestEffort. 
       koordinator.sh/qosClass: "BE"
   spec:
     containers:
     - resources:
         requests:
           # Unit: millicores. In the following example, the CPU request is set to one core. 
           kubernetes.io/batch-cpu: "1k"
           # Unit: bytes. In the following example, the memory request is set to 1 GB. 
           kubernetes.io/batch-memory: "1Gi"
         limits:
           kubernetes.io/batch-cpu: "1k"
           kubernetes.io/batch-memory: "1Gi"

View the usage of Batch resources in Managed Service for Prometheus

ACK clusters are integrated with Managed Service for Prometheus, which provides Prometheus dashboards. You can view the usage of Batch resources in the ACK console.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Operations > Prometheus Monitoring.

3. Choose Others > k8s-reclaimed-resource.

   On the k8s-reclaimed-resource tab, you can view details about the Batch resources. The details include the total amount of Batch resources provided and requested by each node and the cluster. For more information, see Basic monitoring capabilities.

   If you created a Prometheus dashboard, you can view the data of colocated resources based on the following metrics:

   # The amount of allocatable batch-cpu resources on the node. 
   koordlet_node_resource_allocatable{resource="kubernetes.io/batch-cpu",node="$node"}
   # The amount of batch-cpu resources that are allocated on the node. 
   koordlet_container_resource_requests{resource="kubernetes.io/batch-cpu",node="$node"}
   # The amount of allocatable batch-memory resources on the node. 
   kube_node_status_allocatable{resource="kubernetes.io/batch-memory",node="$node"}
   # The amount of batch-memory resources that are allocated on the node. 
   koordlet_container_resource_requests{resource="kubernetes.io/batch-memory",node="$node"}

Is the resource overcommitment feature that is enabled based on the earlier version of the ack-slo-manager protocol supported after I upgrade from ack-slo-manager to ack-koordinator?

The earlier version of the ack-slo-manager protocol includes the following components:

• The alibabacloud.com/qosClass pod annotation.

• The alibabacloud.com/reclaimed field that is used to specify the resource requests and limits of pods.

ack-koordinator is compatible with the earlier version of the ack-slo-manager protocol. The scheduler of an ACK Pro cluster can calculate the amount of requested resources and the amount of available resources based on the earlier protocol version and the new protocol version. You can seamlessly upgrade from ack-slo-manager to ack-koordinator.

The following table describes the compatibility between the scheduler of an ACK Pro cluster, ack-koordinator, and different protocols.

Why does the memory usage suddenly increase after an application uses Batch resources?

For applications configured with kubernetes.io/batch-memory (Batch memory limit), ack-koordinator specifies the memory limit in the cgroup of the node based on the Batch memory limit after the container is created. Some applications automatically request memory based on the cgroup of the container when the applications are started. If an application is started before the memory limit of the cgroup is configured, the actual memory usage may exceed the Batch memory limit. However, the operating system does not immediately reduce the memory usage of the process. As a result, the memory limit in the cgroup of the container cannot be specified until the actual usage falls below the Batch memory limit.

In this case, we recommend that you modify the application configuration to ensure that the actual memory usage remains below the Batch memory limit. You can also check the memory limit parameters in the application boot script to ensure that the parameters are configured before the application starts. This ensures that the memory usage of the application is properly limited and avoids OOM errors.

Run the following command in the container to query the memory limit:

# Unit: bytes. 
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 
# The expected output. 
1048576000