Alibaba Cloud Linux:Use KFENCE to detect kernel memory pollution

Scenario 1: Use KFENCE to detect whether a memory pollution issue occurs

• Add the kfence.sample_interval parameter to enable KFENCE.

  Replace <kfence.sample_interval> with the value that you want to specify. For example, a value of 100 specifies that the KFENCE debugging tool is automatically enabled the next time the system starts and the sampling interval is set to 100 events.

  sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="kfence.sample_interval=<kfence.sample_interval>"

• Add the kfence.booting_max parameter to limit the maximum amount of memory that KFENCE can consume based on the memory specifications.

  Note

  • In kernel version 5.10.134-17 or later, the default configuration kfence.booting_max=0-2G:0,2G-32G:2M,32G-:32M is added to the boot commandline parameter list. The preceding default configuration is used in conjunction with the default value (255) of the num_objects parameter to ensure that the memory overhead of KFENCE does not exceed 1‰ of the total memory in all memory specifications. When the preceding default configuration and value are used, KFENCE can consume up to 2 MiB of memory if standard 4-KiB memory pages are used and up to 32 MiB of memory if 64-KiB huge pages are used.

  • The kfence.booting_max parameter only limits the maximum amount of memory that KFENCE can consume. The parameter is a constraint for the num_objects parameter and does not represent the actual memory overhead. The actual memory overhead is less than or equal to the value of the kfence.booting_max parameter.

  Replace <kfence.booting_max> with the value that you want to specify, such as 0-128M:0,128M-256M:1M,256M-:2M. Description of the segments in the sample value:

  • 0-128M:0: If the total memory on the machine that you use is less than 128 MiB in size, KFENCE is disabled.

  • 128M-256M:1M: If the total memory on the machine that you use is larger than or equal to 128 MiB but less than or equal to 256 MiB in size, KFENCE can consume up to 1 MiB of memory. The value of the num_objects parameter cannot exceed 127.

  • 256M-:2M: If the total memory on the machine that you use is larger than 256 MiB in size, KFENCE can consume up to 2 MiB of memory. The value of the num_objects parameter cannot exceed 255.

  sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="kfence.booting_max=<kfence.booting_max>"

  The preceding configuration applies only to the scenario in which KFENCE is started on system startup by adding parameters to the boot commandline parameter list. The configuration does not take effect in Scenario 2 in which KFENCE is configured after the system starts.

The configuration automatically takes effect the next time the system starts.

Scenario 2: Use KFENCE to detect whether a memory pollution issue occurs

1. Create a memory allocation script and add the following content. In the following example, the script is named kfence.sh and the slab type to be monitored is kmalloc-64.

   #!/bin/bash
   # usage: ./kfence.sh kmalloc-64
   
   SLAB_PREFIX=/sys/kernel/slab
   MODULE_PREFIX=/sys/module/kfence/parameters
   
   if [ $# -eq 0 ]; then
   	echo "err: please input slabs"
   	exit 1
   fi
   
   #check whether slab exists
   for i in $@; do
   	slab_path=$SLAB_PREFIX/$i
   	if [ !  -d $slab_path ]; then
   		echo "err: slab $i not exist!"
   		exit 1
   	fi
   done
   
   #calculate num_objects
   sumobj=0
   for i in $@; do
   	objects=($(cat $SLAB_PREFIX/$i/objects))
   	maxobj=1
   	for ((j=1; j<${#objects[@]}; j++)); do
   		nodeobj=$(echo ${objects[$j]} | awk -F= '{print $2}')
   		[ $maxobj -lt $nodeobj ] && maxobj=$nodeobj
   	done
   	((sumobj += maxobj))
   done
   echo "recommend num_objects per node: $sumobj"
   
   #check kfence stats
   if [ $(cat $MODULE_PREFIX/sample_interval) -ne 0 ]; then
   	echo "kfence is running, disable it and wait..."
   	echo 0 > $MODULE_PREFIX/sample_interval
   	sleep 1
   fi
   
   #disable all slabs catching
   for file in $SLAB_PREFIX/*
   do
   	(echo 0 > $file/kfence_enable) 2>/dev/null || echo 1 > $file/skip_kfence
   done
   
   #disable order0 page catching
   echo 0 > $MODULE_PREFIX/order0_page
   
   #enable setting slabs catching
   for i in $@; do
   	(echo 1 > $SLAB_PREFIX/$i/kfence_enable) 2>/dev/null || echo 0 > $SLAB_PREFIX/$i/skip_kfence
   done
   
   #setting num_objects and node mode
   echo $sumobj > $MODULE_PREFIX/num_objects
   echo node > $MODULE_PREFIX/pool_mode
   
   #start kfence
   echo -1 > $MODULE_PREFIX/sample_interval
   if [ $?  -ne 0 ]; then
   	echo "err: kfence enable fail!"
   	exit 1
   fi
   echo "kfence enabled!"

   The script is used to detect the number of active objects of the slabs, estimate the appropriate KFENCE pool size based on the number, and then enable KFENCE to obtain information about the memory allocation of all the slabs.

   Note

   Slabs are commonly used in memory management to optimize memory allocation and release operations. This improves system performance and efficiency. KFENCE can monitor slabs and order-0 pages.

2. Run the following command to execute the script to start memory pollution detection:

   sudo bash ./kfence.sh kmalloc-64