All Products
Search
Document Center

Alibaba Cloud Linux:Group identity feature

Last Updated:Sep 25, 2024

Alibaba Cloud Linux 2 with kernel 4.19.91-24.al7 or later and Alibaba Cloud Linux 3 with kernel 5.10.46-7.al8 or later support the group identity feature. You can use the group identity feature to configure different identities for CPU control groups (cgroups) to define the priorities of process tasks in the cgroups.

Prerequisites

Note
  • Alibaba Cloud Linux 2 with kernel 4.19.91-26, 4.19.91-26.1, 4.19.91-26.2, or 4.19.91-26.3 does not support the group identity feature because the feature is disabled in the kernel. You can run the uname -r command to view the kernel version of Alibaba Cloud Linux 2.

  • Alibaba Cloud Linux 3 with kernel 5.10.112-11.al8, 5.10.112-11.1.al8, 5.10.112-11.2.al8, 5.10.134-12.al8, 5.10.134-12.1.al8, or 5.10.134-12.2.al8 does not support the group identity feature because the feature is disabled in the kernel. You can run the uname -r command to view the kernel version of Alibaba Cloud Linux 3.

  • If you use the group identity feature on Alibaba Cloud Linux 2 with a kernel version in the range of 4.19.91-25.1.al7to 4.19.91-25.5.al7, downtime occurs. Before you use the group identity feature, upgrade the kernel version to 4.19.91-25.6.al7 or later. For more information, see the FAQ section of this topic.

  • If Alibaba Cloud Linux 3 whose kernel version is 5.10.134-12.2.al8 uses the x86_64 architecture, run the following commands to use the group identity feature:

    yum makecache
    sudo yum install scheduler-group-identity.x86_64 -y
  • In Alibaba Cloud Linux 2 with kernel 4.19.91-26.4 or later and Alibaba Cloud Linux 3 with kernel 5.10.134-13.al8 or later, the /proc/sys/kernel/sched_group_identity_enabled interface is added to allow you to enable the group identity feature. Before you can use the group identity feature, you must run the echo 1 > /proc/sys/kernel/sched_group_identity_enabled command to enable the feature.

Background information

When you deploy latency-sensitive tasks and computing tasks on the same instance, the Linux kernel scheduler must provide more scheduling opportunities to high-priority tasks to minimize scheduling latency and the impacts of low-priority tasks on kernel scheduling. In the preceding scenario, Alibaba Cloud Linux provides the group identity feature and adds interfaces that can be used to configure scheduling priorities for CPU cgroups. Tasks that have different priorities have the following characteristics:

  • High-priority tasks have the minimum wakeup latency.

  • Low-priority tasks do not affect the performance of high-priority tasks.

    • The wakeup of low-priority tasks does not affect the performance of high-priority tasks.

    • Low-priority tasks do not affect the performance of high-priority tasks by sharing hardware units.

How the group identity feature works

The group identity feature allows you to configure identities for CPU cgroups to define the priorities of tasks in the cgroups. The group identity feature relies on a dual red-black tree architecture. A low-priority red-black tree is added based on the red-black tree of the Completely Fair Scheduler (CFS) scheduling queue to store low-priority tasks.

When the kernel schedules the tasks for which identities are configured, the kernel processes the tasks based on their priorities. The following table describes the identities in descending order of priority.

Identity

Description

ID_HIGHCLASS

Identifies a high-priority task. A high-priority task has more opportunities to preempt resources than a normal- or low-priority task.

When the CFS schedules high-priority tasks, the following scenarios may occur:

  • If a high-priority task wakes up when a low-priority task is running, the high-priority task can unconditionally preempt resources from the low-priority task.

  • If a high-priority task wakes up when a normal-priority task is running and the virtual runtime (vruntime) of the high-priority task is shorter than the vruntime of the normal-priority task, the high-priority task can ignore the original scheduling policy and preempt resources. The original scheduling policy specifies that a task cannot preempt resources when the runtime of the task on a CPU is shorter than the minimum runtime.

  • If a low- or normal-priority task is running when tasks queue up to run, a high-priority task whose vruntime is shorter than the vruntime of the running task can ignore the original scheduling policy and preempt resources. The original scheduling policy specifies that a task cannot preempt resources when the runtime of the task on a CPU is shorter than the minimum runtime.

ID_NORMAL

Identifies a normal-priority task. A normal-priority task has more opportunities to preempt resources than a low-priority task.

When the CFS schedules normal-priority tasks, the following scenarios may occur:

  • If a normal-priority task wakes up when a low-priority task is running, the normal-priority task can unconditionally preempt resources from the low-priority task.

  • If a low-priority task is running when tasks queue up to run, a normal-priority task whose vruntime is shorter than the vruntime of the running task can ignore the original scheduling policy and preempt resources. The original scheduling policy specifies that a task cannot preempt resources when the runtime of the task on a CPU is shorter than the minimum runtime.

ID_UNDERCLASS

Identifies a low-priority task.

When the CFS schedules low-priority tasks, the following scenarios may occur:

If an ID_SMT_EXPELLER task runs on a simultaneous multithreading (SMT) CPU, low-priority tasks are evicted from the peer SMT CPU.

The preceding identities apply based on the resource management policies of CPU cgroups.

  • For tasks in CPU cgroups of the same level, identity priorities take effect.

  • For tasks in CPU cgroups of different levels, identity priorities do not take effect on tasks in parent cgroups but take effect on tasks in child cgroups.

  • For tasks that have the same identity priority, resources are competed for in compliance with CFS policies. Take note that the runtime of tasks identified by the ID_UNDERCLASS or ID_NORMAL identity may not reach the minimum value.

Other identities

Identity

Description

ID_SMT_EXPELLER

Identifies an SMT expeller task. When an SMT expeller task runs an SMT CPU, the tasks that are identified by the ID_UNDERCLASS identity are evicted from the peer SMT CPU.

ID_IDLE_SEEKER

Specifies that when a task wakes up, the task attempts to find idle CPUs within the limits of scheduler policies.

ID_IDLE_SAVER

Used with the sched_idle_saver_wmark kernel parameter. You can use the sched_idle_saver_wmark kernel parameter to configure a watermark for CPU idle time. When a task identified by the ID_IDLE_SAVER identity wakes up, the task attempts to find a CPU whose idle time exceeds the specified watermark.

Interfaces

  • Interfaces used to configure identities

    The group identity feature provides the following interfaces for you to configure task identities: /sys/fs/cgroup/cpu/$cg/cpu.identity and /sys/fs/cgroup/cpu/$cg/cpu.bvt_warp_ns. The $cg variable specifies the child cgroup directory node on which a task is located. Before you use the interfaces, take note of the following items:

    • The cpu.bvt_warp_ns interface is a quick configuration interface. The written value of the interface is converted into identities.

    • You can use the cpu.identity and cpu.bvt_warp_ns interfaces to change the identities of CPU cgroups.

    • The identity value that is written by using the cpu.identity interface overwrites the identity value that is previously written by using the cpu.bvt_warp_ns interface, but the value of the cpu.bvt_warp_ns interface remains unchanged.

    • The identity value that is written by using the cpu.bvt_warp_ns interface overwrites the identity value that is previously written by using the cpu.identity interface, but the value of the cpu.identity interface remains unchanged.

    • You can use one of the interfaces to configure task identities. We recommend that you do not use the interfaces at the same time.

    • If you are unfamiliar with the operations related to the operating system kernel, we recommend that you do not use the cpu.identity interface.

    The following table describes the interfaces.

    Interface

    Description

    cpu.identity

    The default value is 0, which specifies the ID_NORMAL identity.

    The interface is a 5-bit field. Valid values of each bit: 0 and 1. 0 specifies that the identity is not assumed. 1 specifies that the identity is assumed. Description of each bit:

    • If the interface is left empty, the ID_NORMAL identity is used.

    • Bit 0: specifies the ID_UNDERCLASS identity.

    • Bit 1: specifies the ID_HIGHCLASS identity.

    • Bit 2: specifies the ID_SMT_EXPELLER identity.

    • Bit 3: specifies the ID_IDLE_SAVER identity.

    • Bit 4: specifies the ID_IDLE_SEEKER identity.

    For example, if you want to set the identity of a cgroup to ID_HIGHCLASS and ID_IDLE_SEEKER, set bit 1 and bit 4 to 1 and the other bits to 0 to obtain a binary value of 10010, which is converted into a decimal value of 18. Then, run the echo 18 > /sys/fs/cgroup/cpu/$cg/cpu.identity command to write 18 to the cpu.identity interface.

    cpu.bvt_warp_ns

    The default value is 0, which specifies the ID_NORMAL identity. Valid values:

    • 2: specifies the ID_SMT_EXPELLER, ID_IDLE_SEEKER, and ID_HIGHCLASS identities. The corresponding value in the cpu.identity interface is 22.

    • 1: specifies the ID_HIGHCLASS and ID_IDLE_SEEKER identities. The corresponding value in cpu.identity is 18.

    • 0: specifies the ID_NORMAL identity. The corresponding value in the cpu.identity interface is 0.

    • -1: specifies the ID_UNDERCLASS and ID_IDLE_SAVER identities. The corresponding value in the cpu.identity interface is 9.

    • -2: specifies the ID_UNDERCLASS and ID_IDLE_SAVER identities. The corresponding value in the cpu.identity interface is 9.

    Note

    By default, Alibaba Cloud Linux supports the cgroup v1 interfaces. Alibaba Cloud Linux 3 with kernel 5.10.134-13 and later in the 5.10 kernel series also supports the following cgroup v2 interfaces for the group identity feature:

    /sys/fs/cgroup/$cg/cpu.identity and /sys/fs/cgroup/$cg/cpu.bvt_warp_ns. The $cg variable specifies the child cgroup directory node on which a task is located.

  • Interfaces used to enable or disable kernel scheduling features

    You can run the following command to view the default settings of kernel scheduling features by using the sched_features interface:

    sudo cat /sys/kernel/debug/sched_features

    The following table describes the scheduling features.

    Scheduling feature

    Description

    Default value

    ID_IDLE_AVG

    This feature is used together with the ID_IDLE_SAVER identity to count the runtime of ID_UNDERCLASS tasks towards the idle time. This ensures that no CPUs remain idle when only ID_UNDERCLASS tasks are running, and prevents resource waste.

    ID_IDLE_AVG: indicates that the feature is enabled.

    ID_RESCUE_EXPELLEE

    This feature is used in load balancing scenarios. If tasks cannot find available CPU resources, CPUs that are evicting ID_UNDERCLASS tasks are used for balancing loads. This feature helps move ID_UNDERCLASS tasks out of the evicted state at the earliest opportunity.

    ID_RESCUE_EXPELLEE: indicates that the feature is enabled.

    ID_EXPELLEE_NEVER_HOT

    After this feature is enabled, if a request is initiated to migrate a task that is being evicted to another CPU, the migration request is not denied due to hot cache. This feature helps move ID_UNDERCLASS tasks out of the evicted state at the earliest opportunity.

    NO_ID_EXPELLEE_NEVER_HOT: indicates that the feature is disabled.

    ID_LOOSE_EXPEL

    After this feature is enabled, CPUs do not update the eviction status every time the CPUs select tasks but have the status automatically updated at the time specified by the sched_expel_update_interval kernel parameter. The configuration of this feature affects only status updates when CPUs select tasks. Updates of inter-processor interrupts (IPIs) are not affected.

    NO_ID_LOOSE_EXPEL: indicates that the feature is disabled.

    ID_LAST_HIGHCLASS_STAY

    After this feature is enabled, the last ID_HIGHCLASS task that runs on a CPU cannot be migrated to another CPU.

    ID_LAST_HIGHCLASS_STAY: indicates that the feature is enabled.

    ID_EXPELLER_SHARE_CORE

    • If this feature is enabled, ID_SMT_EXPELLER tasks can preferentially run on physical cores on which ID_SMT_EXPELLER tasks are already running.

    • If this feature is disabled, ID_SMT_EXPELLER tasks are distributed across physical cores. This way, the ID_SMT_EXPELLER tasks do not interfere with each other.

    ID_EXPELLER_SHARE_CORE: indicates that the feature is enabled.

    ID_ABSOLUTE_EXPEL

    In Alibaba Cloud Linux 3, this feature is introduced in kernel 5.10.134-16.3 and is usable in kernel 5.10.134-16.3 and later in the 5.10 kernel series. After this feature is enabled, ID_UNDERCLASS tasks are absolutely suppressed and cannot be scheduled if ID_NORMAL or ID_HIGHCLASS tasks are in the task queues for running. In worst case scenarios, ID_UNDERCLASS tasks starve. In hybrid deployment scenarios, assess the loads of tasks that have different identities before you enable the feature.

    NO_ID_ABSOLUTE_EXPEL: indicates that the feature is disabled.

    ID_LOAD_BALANCE

    In Alibaba Cloud Linux 3, this feature is introduced in kernel 5.10.134-16.3 and is usable in kernel 5.10.134-16.3 and later in the 5.10 kernel series. After this feature is enabled, when a scheduler balances loads, the scheduler considers the CPUs on which only ID_UNDERCLASS tasks run to be idle and attempts to migrate ID_HIGHCLASS tasks to the idle CPUs. During the migration, the scheduler tries to distribute the ID_HIGHCLASS tasks across the CPUs. This prevents CPU resource contention and Hyper-Threading (HT) inference between the ID_HIGHCLASS tasks and ensures that each ID_HIGHCLASS task can obtain sufficient CPU resources.

    NO_ID_LOAD_BALANCE: indicates that the feature is disabled.

  • Interfaces used by sysctl to configure kernel parameters

    Some capabilities of the group identity feature depend on the values of kernel parameters. The following table describes the kernel parameters.

    Kernel parameter

    Description

    Unit

    Default value

    /proc/sys/kernel/sched_expel_update_interval

    The interval at which the eviction status is automatically updated when a CPU selects tasks. This kernel parameter takes effect only the ID_LOOSE_EXPEL feature is enabled.

    ms

    10

    /proc/sys/kernel/sched_expel_idle_balance_delay

    The minimum idle balance interval when a CPU is evicting tasks. A value of -1 specifies that idle balance is not allowed.

    If only ID_UNDERCLASS tasks exist on a CPU and the tasks are being evicted, the CPU is idle. idle balance is performed on the CPU to improve load-balancing effects. However, this may damage ID_UNDERCLASS tasks. You can configure the sched_expel_idle_balance_delay parameter to alleviate the issue.

    ms

    -1

    /proc/sys/kernel/sched_idle_saver_wmark

    The watermark for CPU idle time. When an ID_IDLE_SAVER task wakes up, the task attempts to find an idle CPU whose idle time exceeds the specified watermark.

    ns

    0

    /proc/sys/kernel/sched_group_identity_enabled

    In kernel 4.19.91-26.4 and later, the /proc/sys/kernel/sched_group_identity_enabled interface is added to allow you to enable the group identity feature. Before you can use the group identity feature, you must run the echo 1 > /proc/sys/kernel/sched_group_identity_enabled command to enable the feature.

    After the group identity feature is enabled, data cannot be written to the /proc/sys/kernel/sched_group_identity_enabled interface if the value of the cpu.bvt_warp_ns or cpu.identity interface of the cgroup is not zero.

    Note

    If your kernel version is 4.19.91-26.4.al7, 4.19.91-26.5.al7, or 4.19.91-26.6.al7, the sched_group_identity_enabled interface is set to 1, and the value of the cpu.bvt_warp_ns interface of the cgroup is not zero, errors occur when you read the /proc/sys/kernel/sched_group_identity_enabled settings. This is a read bug that does not affect the normal usage of the interface. This bug is fixed in kernel 4.19.91-27.al7 and later.

    N/A

    0

Information output

When you use the group identity feature, you can run the following command to view various parameters:

cat /proc/sched_debug

The following table describes the output parameters.

Parameter

Description

nr_high_running

The number of ID_HIGHCLASS tasks that are running on the current CPU.

nr_under_running

The number of ID_UNDERCLASS tasks that are running on the current CPU.

nr_expel_immune

The number of non-ID_UNDERCLASS tasks that are running on the current CPU.

smt_expeller

Indicates whether ID_SMT_EXPELLER tasks are running on the current CPU. A value of 1 indicates that ID_SMT_EXPELLER tasks are running on the current CPU. A value of 0 indicates that no ID_SMT_EXPELLER tasks are running on the current CPU.

on_expel

Indicates whether ID_SMT_EXPELLER tasks are running on the peer SMT CPU. A value of 1 indicates that ID_SMT_EXPELLER tasks are running on the peer SMT CPU. A value of 0 indicates that no ID_SMT_EXPELLER tasks are running on the peer SMT CPU.

high_exec_sum

The cumulative runtime of ID_HIGHCLASS tasks on the current CPU.

under_exec_sum

The cumulative runtime of ID_UNDERCLASS tasks on the current CPU.

h_nr_expel_immune

The number of non-ID_UNDERCLASS tasks that are running on cfs_rq.

expel_start

The difference between the minimum vruntimes of the two red-black trees when the CPU starts to evict tasks.

expel_spread

The cumulative difference between the minimum vruntimes of the two red-black trees caused by CPU eviction status.

min_under_vruntime

The minimum vruntime of the low-priority red-black tree.

FAQ

Question: How do I upgrade a kernel version in the range of 4.19.91-25.1.al7 to 4.19.91-25.5.al7 to 4.19.91-25.6.al7 or later?

Answer: Perform the following steps:

  1. Log on to the Elastic Compute Service (ECS) instance whose kernel version you want to upgrade.

    For more information, see Connect to a Linux instance by using a password or key.

  2. Query the kernel version:

    uname -r
  3. Upgrade the latest kernel version:

    sudo yum update kernel -y
  4. Run the following command to restart the ECS instance for the new kernel version to take effect:

    sudo reboot