Disable the estimation feature of IPVS to prevent network jitters - Alibaba Cloud Linux

IP Virtual Server (IPVS) collects per-service statistics through a built-in timer named estimation_timer. In large-scale environments, this timer causes network latency of tens to hundreds of milliseconds and triggers jitters. You can disable the estimation feature on Alibaba Cloud Linux to eliminate these overheads.

How the estimation feature causes jitters

IPVS

IPVS is a Linux kernel component that directs TCP and UDP requests to real servers, providing kernel-mode Layer 4 Server Load Balancer (SLB) operations. In Kubernetes, IPVS serves as a proxy for the kube-proxy component, forwarding service requests to backend Pods for load balancing.

Note

kube-proxy is a Kubernetes cluster component that provides network proxy and load balancing for services.

The estimation feature of IPVS and the cause of network request latency or jitters

The estimation_timer counts the number of connections and data packets sent and received by each service per time slice. The timer is enabled by default. It causes network jitters for two reasons:

Linear scaling with services and CPUs. The kernel stores statistical data on a per-CPU basis. As services grow, computation increases linearly. As CPUs grow, the kernel must traverse more CPUs to collect statistics. When both are numerous, the overhead of each estimation_timer cycle increases significantly.
Timer priority blocks other tasks. In Linux, timer execution has high priority. While a timer runs, other tasks cannot execute. This delays packet reception, causes latency of tens to hundreds of milliseconds for application requests, and triggers jitters during ping operations.

Supported kernel versions

You can disable the IPVS estimation feature only on the following Alibaba Cloud Linux kernel versions:

Distribution	Minimum kernel version
Alibaba Cloud Linux 2	`4.19.91-22`
Alibaba Cloud Linux 3	`5.10.134-14`

Disable the estimation feature

Important

Disabling the estimation feature makes IPVS service statistics unavailable. Commands such as sudo ipvsadm -Ln --stats will no longer report connection count or received packet count changes. Evaluate the impact on your workloads before proceeding.

Connect to your Elastic Compute Service (ECS) instance.
For more information, see Connect to a Linux instance by using a password or key.
Configure the sysctl parameter.
1. Open the sysctl configuration file:
```
sudo vim /etc/sysctl.conf
```
2. Add the following line at the end of the file, then save and close:
```
net.ipv4.vs.run_estimation = 0
```
3. Apply the configuration:
```
sudo sysctl -p
```
Handle container scenarios.
In container environments, perform the following additional steps.
- IPVS module reloads
  Each time the IPVS module reloads, you must rerun sysctl -p for the configuration to take effect. To automate this, create a modprobe configuration file:
  1. Create the file:
```
sudo vim /etc/modprobe.d/ipvs.conf
```
  2. Add the following content, then save and close:
```
install ip_vs /sbin/modprobe --ignore-install ip_vs && /sbin/sysctl -p
```
  This ensures that the sysctl.conf settings apply automatically when the system reloads the IPVS module.
- Network namespace containers
  If you configured IPVS rules for containers that use network namespaces, disable the estimation feature in each container separately.
  Note
  For Alibaba Cloud Container Service for Kubernetes (ACK), the default kube-proxy proxy mode is IPVS, and IPVS rules are configured on the host side. ACK users do not need to disable estimation for each container. Skip this step if you use ACK.
Verify that the feature is disabled.
Use either of the following methods to confirm that the estimation feature is disabled.
- Method 1: Check the sysctl parameter. Run the following command. If the output is 0, the feature is disabled.
```
sudo sysctl net.ipv4.vs.run_estimation
```
- Method 2: Check IPVS statistics. Run the following command multiple times. If the statistical values do not change between runs, the feature is disabled.
  Important
  If ipvsadm is not installed, install it first:
```
sudo yum install ipvsadm
```
```
sudo ipvsadm -Ln --stats
```

What to do next

Verify that network jitter and latency issues are resolved

After you confirm the feature is disabled, verify that jitters and latency are resolved:

Ping test. Ping the IP address or domain name of the ECS instance. If the latency remains stable and within a reasonable range, the issue is resolved.
Performance test. Use a network performance testing tool to run a latency and jitter test. If long tail latency is significantly reduced or no longer occurs, the issue is resolved.
Note
If jitters or latency persist after you disable the estimation feature, use the network diagnostics feature in the Container Service for Kubernetes (ACK) console to investigate. For more information, see Network diagnostics.

Re-enable the estimation feature

To re-enable estimation, change the run_estimation value to 1 in both the sysctl configuration file (/etc/sysctl.conf) and the modprobe configuration file (/etc/modprobe.d/ipvs.conf). Then run sudo sysctl -p to apply the change.