Alibaba Cloud Linux:Code duptext feature

Background information

• Code snippet cross-node access and impacts on the system

  • Code snippet cross-node access

    NUMA is a computer architecture that is designed to resolve issues such as memory access latency and bandwidth load imbalance in multiprocessor systems. In the NUMA architecture, a node consists of processor cores and memory subsystems. Each node has local memory. One NUMA node may run an application or process whose code snippets are stored on another NUMA node. This occurs because the operating system may allocate memory to multiple nodes to resolve insufficient memory issues or meet load balancing requirements. In the preceding cross-node access scenario, the application or process must obtain instructions from the code snippets that are stored on another node.

  • Impacts on the system

    Cross-node access to code snippets may cause specific performance issues, such as increased access latency, reduced execution efficiency, and increased memory bandwidth usage. The preceding issues occur because cross-node access requires data transfer over a bus or interconnection network and memory access to remote nodes. These operations are slower than the operations involved in on-premises memory access. We recommend that you do not perform operations that require cross-node access.

• Principles of the code duptext feature that resolves issues caused by cross-node access to code snippets

  When cross-node access is detected for a code snippet, the code duptext feature creates a copy for the code snippet from a remote node and stores the copy on an on-premises node. This way, an application or process on the on-premises node can access the code snippet copy without cross-node access, which prevents additional latency and memory overheads. The following figure shows the workflow of the code duptext feature.

  1. Process 1 on node 1 accesses a libc.so snippet on node 0, or process 0 on node 0 accesses a test snippet on node 1.

  2. The code duptext feature is used to create a libc.so copy on node 1 and a test copy on node 0.

  3. Process 1 can access the libc.so copy, and process 0 can access the test copy, which eliminates cross-node access.

  Note

  The kernel manages memory at the page-level granularity. Each page is 4 KB in size. The code snippets of a program are stored on one or more memory pages. The first time the code snippets of a program are read into memory from a hard disk and added to the page cache of a memory page, the page on which the code snippets are stored is called the main page. If the kernel detects that a program on an on-premises node accesses code snippets on other nodes, the kernel uses the code duptext feature to create a copy of the code snippets on the on-premises node at the page-level granularity. The page that stores the code snippet copy is called a subpage.

  

Limits

Only the following instance types and images support the code duptext feature:

• Instance type: ECS Bare Metal Instance families. For more information, see Overview.

• Image: Alibaba Cloud Linux 3 images that run kernel version 5.10.112-11 or later.

  Note

  To query the kernel version of an image, run the uname -r command.

Enable or disable the code duptext feature

The code duptext feature can be controlled globally or by using the memcg. The kernel can use the code duptext feature for processes only when both the global switch and memcg switch are turned on to enable the code duptext feature.

Use the code duptext feature

In this example, a test program named test.c is compiled and executed on an ECS instance that has two NUMA nodes.

1. Connect to the ECS instance.

   For more information, see Connect to a Linux instance by using a password or key.

2. (Optional) Run the following command to view information about the NUMA nodes of the ECS instance:

   numactl -H

   Note

   If you do not install the numactl tool, run the sudo yum install numactl command to install the tool.

   The following figure shows that the instance has two NUMA nodes: node 0 and node 1.

   

3. Run the following command to compile the test program and generate an executable file.

   In this example, the source code file of test.c is compiled on node 1, and node 1 generates the page cache of the test file.

   numactl -N 1 -m 1 gcc test.c -o test

4. Run the following command to turn on the global switch for the code duptext feature:

   sudo sh -c 'echo 1 > /sys/kernel/mm/duptext/enabled'

5. Run the following commands to create a memcg directory and enable the duptext feature for the memcg:

   sudo mkdir /sys/fs/cgroup/memory/test
   sudo sh -c 'echo 1 > /sys/fs/cgroup/memory/test/memory.allow_duptext'

6. Run the following command to use the code duptext feature to avoid cross-node access.

   In this example, the cgexec and numactl tools are used to run the executable file named test and bind the process to node 0. In this case, a copy of the test-related code snippet is created on node 0. The test program can access the code snippet copy on node 0, without cross-node access.

   sudo cgexec -g "memory:test" numactl -N 0 -m 0 ./test

   Note

   If you do not install the cgexec tool, run the sudo yum install -y libcgroup-tools command to install the tool.

7. Run the following command to view statistics about subpages of the test program:

   sudo cat /proc/$(pidof test)/smaps

   The following sample command output shows statistics about the subpages of the test program. A code copy of the test program is generated on node 0.

   

   Note

   You can also run the following commands to view the subpage statistics on the instance:

   cat /proc/vmstat | grep -i duptext
   cat /proc/meminfo | grep -i duptext

Disable the code duptext feature

You can disable the code duptext feature based on your business requirements. When the code duptext feature is disabled, the kernel automatically clears all subpages on the instance.

1. Connect to the ECS instance.

   For more information, see Connect to a Linux instance by using a password or key.

2. Run the following command to disable the code duptext feature:

   sudo sh -c 'echo 0 > /sys/kernel/mm/duptext/enabled'

3. Run the following command to verify that the code duptext feature is disabled:

   cat /proc/vmstat | grep -i duptext
   cat /proc/meminfo | grep -i duptext

   The following sample command output shows that all subpages on the instance are cleared, which indicates that the code duptext feature is disabled.