:THP-related performance optimization in Alibaba Cloud Linux

Global configurations

In the Alibaba Cloud Linux kernel, the directory of the THP configuration file is /sys/kernel/mm/transparent_hugepage/enabled. You can use one of the following options:

• always

  THP is globally enabled.

• never

  THP is globally disabled.

• madvise

  THP is enabled only in the memory area that is called by the madvise() system call and is flagged by MADV_HUGEPAGE.

  Note

  When an application is flagged by MADV_HUGEPAGE, the kernel knows that the application requires transparent huge pages for memory allocation.

Defragmentation configurations

In addition to global configurations, the system provides the following THP-related defragmentation configurations.

• THP defragmentation: THP defragmentation can merge small pages in the system into transparent huge pages. This reduces memory fragmentation and improves system performance.

• khugepaged defragmentation: khugepaged is a kernel thread that is used for transparent huge page management and defragmentation. This reduces memory fragmentation and improves system performance. khugepaged monitors transparent huge pages in the system and attempts to merge scattered transparent huge pages into larger pages. This improves memory utilization and performance.

Khugepaged defragmentation manages and defragments existing transparent huge pages. THP defragmentation attempts to merge scattered small pages into larger pages. Both configurations reduce memory fragmentation and improve memory utilization and performance by merging pages. See the following description:

System performance impacts of using THP

THP can increase the hit rate of TLB, reduce the number of PTEs and number of memory accesses. This improves system performance. THP attempts to simplify O&M operations without modifying or configuring applications and provide performance improvements that are imperceptible to users. However, the resources of THP are limited. When the system reaches the bottleneck of THP allocation, you need to use certain mechanisms to ensure that the system runs as expected. In this case, the default configuration may not be suitable for all user scenarios. The default configuration may even affect the system performance in the following scenarios:

• If the THP defragmentation is set to always, direct reclaim or direct compaction are performed when the memory is insufficient, which is the same as the behavior for regular pages (4 KB in size). The system waits for both operations to complete and then allocates pages again. This affects the system performance.

• If the khugepaged defragmentation is set to 1, the memory directory is locked when the khugepaged kernel daemon merges memory. If khugepaged defragmentation is triggered during peak hours of your business, the performance of memory-sensitive applications may be affected.

• If you enable THP and disable THP defragmentation and khugepaged defragmentation, the memory allocation process may consume idle page resources faster than regular pages (4 KB in size). This leads to memory reclaim and compaction operations and earlier system performance degrade.

Configuration recommendations

The impact of THP on system performance varies in different scenarios. You need to adjust the configurations based on the situation of your business, system, and application. See the following examples:

• If your system kernel provides sufficient resources, we recommend that you run the following command to enable the experimental option defer+madvise. This allows the kswapd kernel daemon, kcompactd kernel daemon, and khugepaged kernel daemon to work together as much as possible and achieve a balance between memory management and stable performance.

  sudo bash -c "echo 'defer+madvise' > /sys/kernel/mm/transparent_hugepage/defrag"

• If the CPU utilization of the khugepaged kernel daemon reaches or approaches 100%, you can increase the start interval of the khugepaged kernel daemon, for example, to 30 seconds. See the following sample command:

  sudo sh -c 'echo 30000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs'

  Alternatively, you can directly stop the khugepaged kernel daemon. See the following sample command:

  sudo sh -c 'echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag'

• In scenarios that involves a large number of access requests to the database application, a large number of latency-sensitive applications, or a large number of short-lived allocations, we recommend that you disable the THP feature if system stability is more important than performance. See the following sample command to disable THP when the system is running:

  sudo sh -c 'echo "never" > /sys/kernel/mm/transparent_hugepage/enabled'

  Note

  This command is valid only when the system is running. After the system is restarted, the THP feature is enabled. To permanently disable the THP feature, run the following commands in sequence as the root user to add the parameter for disabling the THP feature in the kernel startup parameters.

  sudo grubby --args="transparent_hugepage=never" --update-kernel="/boot/vmlinuz-$(uname -r)"
  sudo reboot