Prevent application performance jitter caused by cgroups - Alibaba Cloud Linux

Control groups (cgroups) are a Linux kernel feature that can be used to allocate, manage, and monitor resources on a system. If the hierarchies or number of cgroups is not appropriate for the system, application performance may degrade. This topic describes the issue of application performance jitter caused by cgroups and how to prevent it from occurring.

Problem description

In container-related business scenarios, applications on a system occasionally experience an increase in the request latency, and the sys metric (the percentage of the CPU that is occupied by the kernel) reaches 30% or higher in the CPU utilization of the host on which the container is deployed. For example, when you run the top command to view the Linux performance data, the sys metric reaches 30%, as indicated by the sy value in the following command output:

top - 16:21:19 up 22 days,  1:59,  3 users,  load average: 40.02, 30.02, 25.06
Tasks: 193 total,   3 running, 190 sleeping,   0 stopped,   0 zombie
%Cpu(s):  20.0 us,  30.0 sy,  0.0 ni, 45.0 id,  5.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 24130808 total,  2071424 free,  1832512 used, 20226872 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 19737776 avail Mem

Possible causes

When you run the cat /proc/cgroups command to view the states of all cgroups, you find that the number of cgroups corresponding to memory is as high as 2040, as shown in the following command output:

#subsys_name  hierarchy num_cgroups enabled
cpuset  2     200     1
cpu     2     200     1
cpuacct 2     200     1
blkio   6     62      1
memory  10    2040    1
devices 5     59      1

In this case, you can use perf to analyze and identify the cause of the issue.

Run the following command to install perf:
```
yum install perf -y
```
Run the following command to sample and analyze the processes of the system:
```
perf record -a -g sleep 10
```
Run the following command to view the analysis results:
```
perf report
```
A command output similar to the following one is returned:
```
31.04%  [kernel]                                      [k] memcg_stat_show
18.61%  [kernel]                                      [k] memcg_sum_events.isra.22
9.41%  [kernel]                                      [k] mem_cgroup_iter
6.94%  [kernel]                                      [k] css_next_descendant_pre
6.11%  [kernel]                                      [k] _find_next_bit
3.96%  [kernel]                                      [k] mem_cgroup_usage.part.43
1.75%  [kernel]                                      [k] find_next_bit
1.38%  [kernel]                                      [k] mem_cgroup_node_nr_lru_pages
```
The command output indicates that the Linux kernel takes most of its running time processing the memcg_stat_show function. The kernel runs for an extended period of time because the number of cgroups corresponding to memory is excessive and the system takes a long time traversing these cgroups.
In addition, excessive numbers of cgroups corresponding to cpuacct and cpu may also affect the performance of Completely Fair Scheduler (CFS) and the load balancing mechanism of the Linux kernel.

Solutions

When you perform O&M on Linux instances, you can use the following solutions to prevent application performance jitter caused by cgroups:

Create no more than 10 child cgroups within each cgroup.
Create no more than 1,000 cgroups in the system and keep the number as minimal as possible.