Disable the estimation feature of IPVS to prevent network jitters - Alibaba Cloud Linux

By default, the IP Virtual Server (IPVS) module of the Linux operating system enables the estimation feature. In scenarios in which a large-sized server hosts a large number of services, such as a large-scale Kubernetes cluster scenario, the estimation feature may cause a latency of tens to hundreds of milliseconds or network jitters when the server processes network requests. You can disable the estimation feature of IPVS for Alibaba Cloud Linux images. If you do not use the feature to collect statistics such as the number of packets and connections, we recommend that you disable the estimation feature of IPVS to eliminate additional overheads and jitters that occur during statistics collection.

Background information

IPVS

IPVS is a Linux kernel component that can direct TCP and UDP service requests to real servers to implement kernel-mode Layer 4 Server Load Balancer (SLB) operations. In Kubernetes containers, IPVS can serve as a proxy for the Kube-proxy component to forward service requests to backend pods to balance Kubernetes services. In large-scale clusters, IPVS can provide faster response and higher resource utilization to meet inter-service traffic distribution and load balance requirements.

Note

The Kube-proxy is an important component in a Kubernetes cluster and implements the network proxy and load balancing features for services.

The estimation feature of IPVS and the cause of network request latency or jitters

IPVS has a built-in timer named estimation_timer, which is used to count the number of connections and the number of data packets sent and received by each service in each time slice. The timer is enabled by default.

The estimation feature of IPVS causes network request latency or jitters due to the following reasons:

As the number of services increases, the amount of computation increases in a linear manner. To reduce the overheads of computing and storage resources, the kernel stores statistical data on a CPU basis. As the number of CPUs used by the kernel increases, the number of CPUs the kernel must traverse to collect statistics also increases. If the numbers of services and CPUs increase, the statistical overheads of each estimation_timer significantly increase.
In Linux, timer execution is assigned a high priority. When a timer task is executed, other tasks cannot be executed. This results in a delay in packet reception. A latency of tens to hundreds of milliseconds occurs when applications send requests to the server. In this case, jitters occur during the ping operation on the server.

Limits

You can disable the estimation feature of IPVS only for Alibaba Cloud Linux images that run the following kernel versions:

Alibaba Cloud Linux 2: 4.19.91-22 or later
Alibaba Cloud Linux 3: 5.10.134-14 or later

Disable the estimation feature to resolve the network jitter issue

Important

When you disable the estimation feature of IPVS by using commands such as the sudo ipvsadm -Ln --stats command, the service statistics feature of IPVS becomes unavailable. As a result, you cannot obtain the changes in the number of connections and the number of received packets for the corresponding services. Evaluate the impacts on the corresponding business. Proceed with caution when you disable the estimation feature.

Connect to an Elastic Compute Service (ECS) instance.
For more information, see Connect to a Linux instance by using a password or key.
Disable the estimation feature on the ECS instance.
1. Run the following command to open the sysctl configuration file named /etc/sysctl.conf:
```
sudo vim /etc/sysctl.conf
```
2. Add the following line at the end of the file. Then, save and close the file.
```
net.ipv4.vs.run_estimation = 0
```
3. Run the following command to allow the modified configuration to take effect:
```
sudo sysctl -p
```
4. In container scenarios, you must perform the following operations to disable the estimation feature.
  - Scenario 1: Each time the IPVS module is reloaded, you must run the sysctl -p command to allow the configuration to take effect. We recommend that you use the modprobe configuration file to automatically run the sysctl -p command each time the system reloads the IPVS module. This way, the sysctl.conf configuration takes effect and the estimation feature is disabled.
    1. Create a modprobe configuration file. In the following example, the ipvs.conf file is created.
      sudo vim /etc/modprobe.d/ipvs.conf
    2. Add the following content to the file. Then, save and close the file.
      options ip_vs run_estimation=0 post-up sysctl -p
  - Scenario 2: If you configured IPVS rules for containers such as the network namespaces container, you must disable the estimation feature for each container.
    Note
    For Alibaba Cloud Container Service for Kubernetes (ACK), the default proxy mode of the Kube-proxy is IPVS, and IPVS rules are configured on the host side. In this case, you do not need to disable the estimation feature for each container. If you use ACK, skip this operation.
5. Verify that the estimation feature is disabled.
  - Method 1: Run the sudo sysctl net.ipv4.vs.run_estimation command. If 0 is returned, the estimation feature is disabled.
  - Method 2: Run the sudo ipvsadm -Ln --stats command. If the statistical value does not change, the estimation feature is disabled.
    Important
    If you did not install the ipvsadm tool, run the sudo yum install ipvsadm command to install the tool.

What to do next

Verify that network jitter and latency issues are resolved

Method 1: Ping the IP address or domain name of the ECS instance. If the latency value remains relatively stable and within a reasonable range, the network latency issue is resolved.
Method 2: Use a network performance testing tool to conduct a comprehensive latency and jitter test to check whether the long tail latency exists. If the frequency and duration of the long tail latency are significantly reduced, or the long tail latency does not recur, the network jitter and latency issues are resolved.
Note
If you disable the estimation feature but network jitters or latency still exist, you can use the network diagnostics feature in the Container Service for Kubernetes (ACK) console to diagnose the issues. For more information, see Network diagnostics.

Re-enable the estimation feature

If you want to re-enable the estimation feature, change the run_estimation value in the sysctl and modprobe configuration files to 1.