blk-iocost weight-based throttling - Alibaba Cloud Linux

Blk-iocost weight-based throttling in Alibaba Cloud Linux (Alinux) enhances the standard weight-based disk throttling of the cgroup I/O subsystem (blkcg). This I/O controller allocates bandwidth for I/O operations on a block device based on application or process priority. By configuring weight values, you can limit the I/O bandwidth of specific applications or processes for granular control over disk I/O resources.

Note

cgroup v1 and cgroup v2 are two different versions of the resource management feature in the Linux kernel. In the Alibaba Cloud Linux kernel, blk-iocost supports both cgroup v1 and v2 interfaces, but typically only one version is active in a system. Run the stat -fc %T /sys/fs/cgroup command to check which version is active.

A return value of tmpfs indicates the cgroup v1 interface.
A return value of cgroup2fs indicates the cgroup v2 interface.

Operating system limitations

Alibaba Cloud Linux 2 with kernel version 4.19.81-17 or later
Alibaba Cloud Linux 3

Use the cost.qos interface

The cost.qos file is a read-write interface used to enable or disable blk-iocost and to rate-limit I/O quality of service (QoS) based on latency and weight. This interface file exists only in the root blkcg group, and its full name varies between cgroup versions.

cgroup v1: blkio.cost.qos
cgroup v2: io.cost.qos

Configuration

Each configuration line begins with the device's major and minor numbers in MAJ:MIN format. You can find these numbers by running the lsblk | grep <Cloud_Disk_Name> command. The device numbers are followed by other parameters.

Parameter

Description

enable

Enables or disables blk-iocost.

0: Default. Disables the blk-iocost feature.
1: Enables the blk-iocost feature.

ctrl

The control mode.

auto: The system automatically detects the device type and uses built-in parameters.
Important
When ctrl is set to auto for an Elastic Compute Service (ECS) instance with a solid-state storage device, such as a standard SSD, Enterprise SSD (ESSD), or NVMe SSD local disk, you must manually set the rotational property of the corresponding cloud disk to 0. This lets blk-iocost estimate I/O cost more accurately and adjust its scheduling policies. Example:
```
sudo sh -c 'echo 0 > /sys/block/<DISK_NAME>/queue/rotational'
```
Replace <DISK_NAME> with the actual disk name.
user: You must manually set the following control parameters:
- rpct: the read latency percentile. Valid values: 0 to 100.
- rlat: the read latency threshold. Unit: microseconds.
- wpct: the write latency percentile. Valid values: 0 to 100.
- wlat: the write latency threshold. Unit: microseconds.
- min: the minimum scaling percentage. Valid values: 1 to 10000.
- max: the maximum scaling percentage. Valid values: 1 to 10000.

Enable the blk-iocost feature

This example enables blk-iocost for device 254:48 in user control mode. In this example, the disk is considered saturated when 95% of read/write request latencies (rlat and wlat) exceed 5000 us (5 ms). The kernel then adjusts the request dispatch rate within a range of 50% to 150%.

cgroup v1 interface

sudo sh -c 'echo "254:48 enable=1 ctrl=user rpct=95.00 rlat=5000 wpct=95.00 wlat=5000 min=50.00 max=150.00" > /sys/fs/cgroup/blkio/blkio.cost.qos'

cgroup v2 interface

sudo sh -c 'echo "254:48 enable=1 ctrl=user rpct=95.00 rlat=5000 wpct=95.00 wlat=5000 min=50.00 max=150.00" > /sys/fs/cgroup/io.cost.qos'

Use the cost.model interface

The cost.model file is a read-write interface used to set the cost model. This interface file exists only in the root blkcg group, and its full name varies between cgroup versions.

cgroup v1: blkio.cost.model
cgroup v2: io.cost.model

Configuration

Parameter

Description

ctrl

The control mode.

auto: The system automatically optimizes I/O scheduling policies based on the current workload.
Important
When ctrl is set to auto for an Elastic Compute Service (ECS) instance with a solid-state storage device, such as a standard SSD, Enterprise SSD (ESSD), or NVMe SSD local disk, you must manually set the rotational property of the corresponding cloud disk to 0. This lets blk-iocost estimate I/O cost more accurately and adjust its scheduling policies. Example:
```
sudo sh -c 'echo 0 > /sys/block/<DISK_NAME>/queue/rotational'
```
Replace <DISK_NAME> with the actual disk name.
user: You must manually enter the model parameters.

model

The model parameter. Currently, only the linear model is implemented. When model is set to linear, you can define the following modeling parameters:

[r|w]bps: the maximum sequential I/O throughput. Unit: bytes/second.
[r|w]seqiops: the sequential input/output operations per second (IOPS) limit.
[r|w]randiops: the random IOPS limit.
Note
You can use the tools/cgroup/iocost_coef_gen.py script in the kernel source code to generate the preceding parameters and then write the parameters to the cost.model interface file to configure the cost model.

Set a cost model

This example sets a cost model for device 254:48 by using user-defined linear model parameters.

cgroup v1 interface

sudo sh -c 'echo "254:48 ctrl=user model=linear rbps=2706339840 rseqiops=89698 rrandiops=110036 wbps=1063126016 wseqiops=135560 wrandiops=130734" > /sys/fs/cgroup/blkio/blkio.cost.model'

cgroup v2 interface

sudo sh -c 'echo "254:48 ctrl=user model=linear rbps=2706339840 rseqiops=89698 rrandiops=110036 wbps=1063126016 wseqiops=135560 wrandiops=130734" > /sys/fs/cgroup/io.cost.model'

Use the weight/cost.weight interface

The name of the core read-write interface for controlling I/O resource allocation depends on the cgroup version and the operating system. By configuring a weight value between 1 and 10000, you can dynamically allocate disk I/O bandwidth. These interface files exist only in blkcg subgroups.

Alibaba Cloud Linux 3
- cgroup v1: blkio.cost.weight
- cgroup v2: io.weight
Alibaba Cloud Linux 2
- cgroup v1: blkio.cost.weight
- cgroup v2: io.cost.weight

Configuration

To modify the default weight of a blkcg, set a weight value for the interface: <weight>.
To modify the weight of a blkcg on a specific device, use the following format: MAJ:MIN <weight>.

Modify weights

After you enable blk-iocost, the following examples create control groups (blkcg1 for cgroup v1 and cg1 for cgroup v2) and set their weights. The examples set the default weight for the control group to 50 and the device-specific weight for device 254:48 to 50.

cgroup v1 interface

sudo mkdir /sys/fs/cgroup/blkio/blkcg1    # Create the control group blkcg1
sudo sh -c 'echo "50" > /sys/fs/cgroup/blkio/blkcg1/blkio.cost.weight'    # Change the default weight to 50
sudo sh -c 'echo "254:48 50" > /sys/fs/cgroup/blkio/blkcg1/blkio.cost.weight'    # Set the weight on the disk to 50

cgroup v2 interface

Alibaba Cloud Linux 3

sudo mkdir /sys/fs/cgroup/cg1    # Create the control group cg1
sudo sh -c 'echo "50" > /sys/fs/cgroup/cg1/io.weight'    # Change the default weight to 50
sudo sh -c 'echo "254:48 50" > /sys/fs/cgroup/cg1/io.weight'    # Set the weight on the disk to 50

Alibaba Cloud Linux 2

sudo mkdir /sys/fs/cgroup/cg1    # Create the control group cg1
sudo sh -c 'echo "50" > /sys/fs/cgroup/cg1/io.cost.weight'    # Change the default weight to 50
sudo sh -c 'echo "254:48 50" > /sys/fs/cgroup/cg1/io.cost.weight'    # Set the weight on the disk to 50

Common monitoring tools

To use blk-iocost effectively, you must monitor and evaluate system I/O performance. The following tools and interfaces help you monitor and tune I/O resource usage.

iocost monitor script

The tools/cgroup/iocost_monitor.py script in the Linux kernel source uses the drgn debugger to directly access kernel parameters and outputs I/O performance monitoring data. To use the script, follow these steps:

Install the drgn debugger.
```
sudo pip3 install drgn
```

Download the iocost_monitor.py script.

wget https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/cgroup/iocost_monitor.py

Run the iocost_monitor.py script.

In this example, the vdd disk is used.

sudo python3 ./iocost_monitor.p vdd

Example output:

vdd RUN  per=500.0ms cur_per=3930.839:v14620.321 busy= +1 vrate=6136.22% params=hdd
                          active    weight      hweight% inflt% dbt  delay usages%
blkcg1                       *    50/   50   9.09/  9.09   0.00   0  0*000 009:009:009
blkcg2                       *   500/  500  90.91/ 90.91   0.00   0  0*000 089:091:092

blkio.cost.stat interface (cgroup v1)
The Alibaba Cloud Linux kernel provides the blkio.cost.stat statistics interface for cgroup v1. This file records QoS data for each controlled device. Run the following command to view the interface file:
```
cat /sys/fs/cgroup/blkio/blkcg1/blkio.cost.stat
```
Example output:
```
254:48 is_active=1 active=50 inuse=50 hweight_active=5957 hweight_inuse=5957 vrate=159571
```

ftrace tool

The Alibaba Cloud Linux kernel provides ftrace-related tools for blk-iocost. For blk-iocost, ftrace helps you capture the scheduler's decision-making process and trace the handling of I/O requests in detail, providing in-depth performance analysis. To use the tool, follow these steps:

Set the enable attribute to 1 to enable the ftrace tool.

sudo sh -c 'echo 1 > /sys/kernel/debug/tracing/events/iocost/enable'

View the output information.

sudo cat /sys/kernel/debug/tracing/trace_pipe

Example output:

    dd-1593  [008] d...   688.565349: iocost_iocg_activate: [vdd:/blkcg1] now=689065289:57986587662878 vrate=137438 period=22->22 vtime=0->57986365150756 weight=50/50 hweight=65536/65536
    dd-1593  [008] d.s.   688.575374: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.608369: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
    dd-1594  [006] d...   688.620002: iocost_iocg_activate: [vdd:/blkcg2] now=689119946:57994099611644 vrate=137438 period=22->26 vtime=0->57993412421644 weight=250/250 hweight=65536/65536
<idle>-0     [008] d.s.   688.631367: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.642368: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.653366: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1
<idle>-0     [008] d.s.   688.664366: iocost_ioc_vrate_adj: [vdd] vrate=137438->137438 busy=0 missed_ppm=0:0 rq_wait_pct=0 lagging=1 shortages=0 surpluses=1