Configure QoS for pods - Container Service for Kubernetes

Online applications are sensitive to the network latency while offline applications consume large amounts of bandwidth resources. If you run both types of applications without limits, unexpected traffic spikes may overwhelm your online applications and adversely affect the entire network. To avoid this issue, you can configure quality of service (QoS) for pods. This topic describes the pod annotations defined by Kubernetes and the relevant bandwidth limits. This topic also describes how to configure QoS classes.

Bandwidth limits

You can use the Terway network plug-in to control the network bandwidth of pods. The following table describes the pod annotations that you can use to specify the inbound and outbound bandwidth of pods.

Annotation	Mean
kubernetes.io/ingress-bandwidth: 10M	Ingress bandwidth
kubernetes.io/egress-bandwidth: 10M	Egress bandwidth

The following table describes features supported in different Terway modes for limiting bandwidth.

Terway Mode	Egress Shaping	Ingress Shaping
VPC mode (legacy)
Shared ENI
Shared ENI + IPVLAN eBPF
Shared ENI + DataPath V2

In the Shared ENI + DataPath V2 mode, pods on nodes that use a kernel version later than 5.10 use Earliest Departure Time (EDT) to limit bandwidth. In other modes, pods use TBF Qdisc to limit bandwidth.

Use pod annotations defined by Kubernetes

In Container Service for Kubernetes (ACK) clusters that use Terway V1.3.0 or later, you can use pod annotations defined by Kubernetes to specify the inbound and outbound bandwidth of pods.
In ACK clusters that use a Terway version earlier than V1.3.0, you can use only the pod annotations defined by Alibaba Cloud to limit the inbound and outbound bandwidth of pods. The following table describes the pod annotations.
Annotation
Mean
k8s.aliyun.com/ingress-bandwidth: 10M
Ingress bandwidth
k8s.aliyun.com/egress-bandwidth: 10M
Egress bandwidth
The pod annotations defined by Alibaba Cloud are deprecated and will soon be removed from ACK. If you want to limit bandwidth, use the pod annotations defined by Kubernetes.
For ACK clusters that use a Terway version earlier than V1.3.0, you need to modify the ConfigMap of Terway to enable the pod annotations defined by Kubernetes.
1. Run the following command to modify the ConfigMap of Terway:
```
kubectl edit cm -n kube-system eni-config
```
2. Add capabilities to the data.10-terway.conf section.
```
apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "capabilities": {"bandwidth": true},
      "type": "terway"
    }
```
3. Press Esc. Then, enter :wq! and press Enter to save the ConfigMap and exit.
4. Run the following command to recreate the Terway pod for the modified ConfigMap to take effect:
```
kubectl delete -n kube-system pod -l app=terway-eniip
```

Configure QoS classes

By default, the elastic network interface (ENI) queue follows the first-in, first-out (FIFO) principle when processing packets. This allows you to prioritize the packets of latency-sensitive applications. By default, traffic prioritization is disabled. You need to manually enable this feature.

The following table describes the pod annotations that can be used to configure QoS classes for pods when Terway is used.

Annotation	Mean
k8s.aliyun.com/network-priority: "guaranteed"	Latency-sensitive applications
k8s.aliyun.com/network-priority: "best-effort"	Regular applications
k8s.aliyun.com/network-priority: "burstable"	Applications that consume large amounts of bandwidth

Perform the following steps to enable traffic prioritization.

Run the following command to modify the ConfigMap of Terway:
```
kubectl edit cm -n kube-system eni-config
```

Add enable_network_priority to the data.10-terway.conf section to enable traffic prioritization.

apiVersion: v1
data:
  10-terway.conf: |
    {
      "cniVersion": "0.3.1",
      "name": "terway",
      "enable_network_priority": true,
      "type": "terway"
    }

After you modify the ConfigMap, press Esc. Then, enter :wq! and press Enter to save the modified file and exit the edit mode.
Recreate the Terway pod for the modified ConfigMap to take effect.
```
kubectl delete -n kube-system pod -l app=terway-eniip
```
Important
After traffic prioritization is enabled, the ENI queue is replaced by mq+prio.

Annotation	Mean
k8s.aliyun.com/ingress-bandwidth: 10M	Ingress bandwidth
k8s.aliyun.com/egress-bandwidth: 10M	Egress bandwidth