Container Service for Kubernetes:Enable memory QoS

Billing

No fee is charged when you install or use the ack-koordinator component. However, fees may be charged in the following scenarios:

• ack-koordinator is a non-managed component that occupies worker node resources after it is installed. You can specify the amount of resources requested by each module when you install the component.

• By default, ack-koordinator exposes the monitoring metrics of features such as resource profiling and fine-grained scheduling as Prometheus metrics. If you enable Prometheus metrics for ack-koordinator and use Managed Service for Prometheus, these metrics are considered custom metrics and fees are charged for these metrics. The fee depends on factors such as the size of your cluster and the number of applications. Before you enable Prometheus metrics, we recommend that you read the Billing topic of Managed Service for Prometheus to learn about the free quota and billing rules of custom metrics. For more information about how to monitor and manage resource usage, see Query the amount of observable data and bills.

Usage notes

When you enable the memory QoS feature for pods, the cgroup parameters are automatically configured based on the specified ratios and pod parameters. This section describes how to enable memory QoS for containers in a specific pod, namespace, or cluster.

annotations:
  # To enable memory QoS for the containers in a pod, set the value to auto. 
  koordinator.sh/memoryQOS: '{"policy": "auto"}'
  # To disable memory QoS for the containers in a pod, set the value to none. 
  koordinator.sh/memoryQOS: '{"policy": "none"}'

Example

In this section, a Redis pod is used as an example. The following conditions are used to compare the latency and throughput of the pod before memory QoS is enabled and after memory QoS is enabled in memory overcommitment scenarios:

• An ACK Pro cluster is used.

• The cluster contains 2 nodes, each of which has 8 vCPUs and 32 GB of memory. One node is used to perform stress tests. The other node runs the workload and serves as the tested machine.

Procedure

1. Create a file named redis-demo.yaml and copy the following content to the file:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: redis-demo-config
   data:
     redis-config: |
       appendonly yes
       appendfsync no
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: redis-demo
     labels:
       koordinator.sh/qosClass: 'LS' # Set the QoS class of the Redis pod to LS. 
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "auto"}' # Add this annotation to enable memory QoS. 
   spec:
     containers:
     - name: redis
       image: redis:5.0.4
       command:
         - redis-server
         - "/redis-master/redis.conf"
       env:
       - name: MASTER
         value: "true"
       ports:
       - containerPort: 6379
       resources:
         limits:
           cpu: "2"
           memory: "6Gi"
         requests:
           cpu: "2"
           memory: "2Gi"
       volumeMounts:
       - mountPath: /redis-master-data
         name: data
       - mountPath: /redis-master
         name: config
     volumes:
       - name: data
         emptyDir: {}
       - name: config
         configMap:
           name: redis-demo-config
           items:
           - key: redis-config
             path: redis.conf
     nodeName: # Set nodeName to the name of the tested node. 
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: redis-demo
   spec:
     ports:
     - name: redis-port
       port: 6379
       protocol: TCP
       targetPort: 6379
     selector:
       name: redis-demo
     type: ClusterIP

2. Run the following command to deploy Redis Server as the test application.

   You can access the redis-demo Service from within the cluster.

   kubectl apply -f redis-demo.yaml

3. Simulate memory overcommitment.

   Use the Stress tool to increase the load on memory and trigger memory reclamation. The sum of the memory limits of all pods on the node exceeds the physical memory of the node.

   1. Create a file named stress-demo.yaml and copy the following content to the file:

      apiVersion: v1
      kind: Pod
      metadata:
        name: stress-demo
        labels:
          koordinator.sh/qosClass: 'BE' # Set the QoS class of the Stress pod to BE. 
        annotations:
          koordinator.sh/memoryQOS: '{"policy": "auto"}' # Add this annotation to enable memory QoS. 
      spec:
        containers:
          - args:
              - '--vm'
              - '2'
              - '--vm-bytes'
              - 11G
              - '-c'
              - '2'
              - '--vm-hang'
              - '2'
            command:
              - stress
            image: polinux/stress
            imagePullPolicy: Always
            name: stress
        restartPolicy: Always
        nodeName: # Set nodeName to the name of the tested node, which is the node on which the Redis pod is deployed.

   2. Run the following command to deploy stress-demo:

      kubectl apply -f stress-demo.yaml

4. Run the following command to query the global minimum watermark of the node:

   Note

   In memory overcommitment scenarios, if the global minimum watermark of the node is set to a low value, OOM killers may be triggered for all pods on the node even before memory reclamation is performed. Therefore, we recommend that you set the global minimum watermark to a high value. In this example, the global minimum watermark is set to 4,000,000 KB for the tested node that has 32 GiB of memory.

   cat /proc/sys/vm/min_free_kbytes

   Expected output:

   4000000

5. Use the following YAML template to deploy the memtier-benchmark tool to send requests to the tested node:

   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       name: memtier-demo
     name: memtier-demo
   spec:
     containers:
       - command:
           - memtier_benchmark
           - '-s'
           - 'redis-demo'
           - '--data-size'
           - '200000'
           - "--ratio"
           - "1:4"
         image: 'redislabs/memtier_benchmark:1.3.0'
         name: memtier
     restartPolicy: Never
     nodeName: # Set nodeName to the name of the node that is used to send requests.

6. Run the following command to query the test results from memtier-benchmark:

   kubectl logs -f memtier-demo

7. Use the following YAML template to disable memory QoS for the Redis pod and Stress pod. Then, perform stress tests again and compare the results.

   apiVersion: v1
   kind: Pod
   metadata:
     name: redis-demo
     labels:
       koordinator.sh/qosClass: 'LS'
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "none"}' # Disable memory QoS. 
   spec:
     ...
   
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     name: stress-demo
     labels:
       koordinator.sh/qosClass: 'BE'
     annotations:
       koordinator.sh/memoryQOS: '{"policy": "none"}' # Disable memory QoS. 
                           

Analyze the result

The following table describes the stress test results when memory QoS is enabled and disabled.

• Disabled: The memory QoS policy of the pod is set to none.

• Enabled: The memory QoS policy of the pod is set to auto and the recommended memory QoS settings are used.

The table shows that the latency of the Redis pod is reduced by 7.9% and the throughput of the Redis pod is increased by 8.7% after memory QoS is enabled. This indicates that the memory QoS feature can optimize the performance of applications in memory overcommitment scenarios.

Advanced parameters

You can enable memory QoS for containers in a specific pod or cluster. If both pod annotations and ConfigMaps are used to configure memory QoS parameters, the pod annotations take precedence. If no pod annotation is added to configure memory QoS, ack-koordinator retrieves memory QoS parameters from ConfigMaps in a specific namespace. If no configuration exists in the ConfigMaps in the namespace, ack-koordinator retrieves memory QoS parameters from ConfigMaps in a specified cluster.

FAQ

Is the memory QoS feature that is enabled based on the earlier version of the ack-slo-manager protocol still supported after I upgrade from ack-slo-manager to ack-koordinator?

In an earlier version (≤ 0.8.0) of the ack-slo-manager protocol, the following pod annotations are used:

• alibabacloud.com/qosClass

• alibabacloud.com/memoryQOS

ack-koordinator is compatible with earlier versions of the ack-slo-manager protocol. You can seamlessly upgrade from ack-slo-manager to ack-koordinator. ack-koordinator is compatible with the earlier protocol versions no later than July 30, 2023. We recommend that you upgrade the resource parameters in an earlier protocol version to the latest version.

The following table describes the compatibility between different versions of ack-koordinator and the memory QoS feature.