Use memory isolation for OOT drivers

0.0.201

important

This topic contains important information on necessary precautions. We recommend that you read this topic carefully before proceeding.

Out of Tree (OOT) drivers vary in quality and often encounter issues, such as user-after-free memory errors and out-of-bounds access. To ensure that corrupted memory is not allocated to other modules and confine damage caused by an OOT driver to the memory allocated to the driver, you must isolate the memory for the driver. This way, you can locate, troubleshoot, and resolve the issue faster and reduce business loss.

Limits

Limits on operating systems
Alibaba Cloud Linux 3 whose kernel version is 5.10.134-17.al8 or later supports memory isolation for OOT drivers.
Limits on functionality
The following functions can be used to allocate 8 bytes to 8 KB of KMALLOC_NORMAL memory for isolation:
- kmalloc
- kzalloc
- kmalloc_node
- kzalloc_node
- kmalloc_array
- kmalloc_array_node
- kcalloc
- kcalloc_node
Limits on quantity
Memory can be isolated for up to 16 OOT drivers.

Configure memory isolation for OOT drivers

You can use one of the following methods to configure memory isolation for OOT drivers based on your business requirements:

Add parameters to configure memory isolation for OOT drivers.
1. In the kernel cmdline, add the following parameter setting: module.oot_list=<OOT module 1>,<OOT module 2>.
  Replace <OOT module 1>,<OOT module 2> with the actual names of OOT modules. If additional OOT modules exist, add the module names in sequence and separate the names with commas (,).
```
sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="module.oot_list=<OOT module 1>,<OOT module 2>"
```
2. Restart the instance for the configurations to take effect.
  Warning
  The restart operation stops the instances for a short period of time and may interrupt the services that are running on the instance. This may result in data loss. Before you restart your instance, we recommend that you back up critical instance data. We also recommend that you restart the instance during off-peak hours.
```
sudo reboot
```
Run the echo command to configure memory isolation for OOT drivers.
1. Write <OOT module 1>,<OOT module 2> to the /sys/module/module/parameters/oot_list file.
  Replace <OOT module 1>,<OOT module 2> with the actual names of OOT modules. If additional OOT modules exist, add the module names in sequence and separate the names with commas (,).
```
sudo sh -c 'echo <OOT module 1>,<OOT module 2> > /sys/module/module/parameters/oot_list'
```
2. Reload the OOT modules for the configurations to take effect.
  Replace <OOT module> with the actual name of an OOT module.
```
rmmod <OOT module>.ko
insmod <OOT module>.ko
```

Note

If you load a memory-isolated OOT module, the following message that contains the index of the separate slab allocated to the module appears in the kernel logs:

Module <Module name> use oot-kmalloc-<index>.

Handle risks associated with memory isolation for OOT drivers

The memory used by OOT drivers is regarded as dirty memory. If the dirty memory is used elsewhere, the dirty memory may be overwritten. By default, memory pages used by an OOT driver are cached for use by the driver and are not returned to the buddy subsystem. This may result in memory overheads. You can use one of the following methods to adjust this behavior:

Configure the module.oot_page_limit parameter to specify the cache usage limit.
1. Obtain the value of the <size> parameter, which specifies the size of a slab. The slab size ranges from 8 B to 8 KB.
```
sudo cat /proc/slabinfo
```
2. Obtain the value of the order parameter.
  Replace <index> and <size> with actual values.
```
/sys/kernel/slab/oot-kmalloc-<index>-<size>/order
```
3. Specify the cache usage limit.
  Replace <limit> with the maximum amount of cache that you want to write. Unit: $2^{or d er}$ pages. For example, if you replace <limit> with 100 and order with 2, the cache usage limit is $100 \times 2^{2}$ pages.
```
sudo grubby --update-kernel=/boot/vmlinuz-$(uname -r) --args="module.oot_page_limit=<limit>"
```
Use the /sys/module/module/parameters/oot_page_limit file to specify the cache usage limit.
1. Obtain the value of the <size> parameter, which specifies the size of a slab. The slab size ranges from 8 B to 8 KB.
```
sudo cat /proc/slabinfo
```
2. Obtain the value of the order parameter.
  Replace <index> and <size> with actual values.
```
/sys/kernel/slab/oot-kmalloc-<index>-<size>/order
```
3. Specify the cache usage limit.
  Replace <limit> with the maximum amount of cache that you want to write. Unit: $2^{or d er}$ pages. For example, if you replace <limit> with 100 and order with 2, the cache usage limit is $100 \times 2^{2}$ pages.
```
sudo sh -c 'echo <limit> > /sys/module/module/parameters/oot_page_limit'
```
Run the /sys/kernel/slab/oot-kmalloc-<index>-<size>/oot_page command to view the unit pages of a specific slab.
1. Obtain the value of the <size> parameter, which specifies the size of a slab. The slab size ranges from 8 B to 8 KB.
```
sudo cat /proc/slabinfo
```
2. View the unit pages of a specific slab.
  Replace <index> and <size> with actual values.
```
/sys/kernel/slab/oot-kmalloc-<index>-<size>/oot_page
```

Enable KFENCE to monitor a specific slab

OOT drivers use separate slabs. You can enable Kernel Electric-Fence (KFENCE) for specific slabs to detect out-of-bound reads and writes. For more information, see Use KFENCE to detect kernel memory pollution.

Run the following commands to enable KFENCE to monitor a specific slab:

#Disable KFENCE from monitoring all slabs.
for file in /sys/kernel/slab/*
do
    sudo sh -c 'echo 0 > $file/kfence_enable'
done

#Disable KFENCE from monitoring order0 pages.
sudo sh -c 'echo 0 > /sys/module/kfence/parameters/order0_page'

#Enable KFENCE to monitor a specific slab.
sudo sh -c 'echo 1 > /sys/kernel/slab/oot-kmalloc-<index>-<size>/kfence_enable'

FAQ

Why does the "`no symbol version for kmalloc/kmalloc_node/kmalloc_caches`" error message appear?

Note

The error message appears because due to non-invasive changes, the system cannot determine whether OOT modules are memory-isolated when the modules are compiled and does not add cyclic redundancy check (CRC) values for OOT-related symbols to the module information. Kernel APIs that use the __kmalloc function are stable and unlikely to be affected by this issue. You can be ignored the error message.

Problem description
The first time a memory-isolated OOT module is loaded, the "no symbol version for __kmalloc/__kmalloc_node/kmalloc_caches" error message appears. The error message appears only once. This is not an issue.
Cause
When the Linux kernel loads a module, the kernel relocates the Undefined (UND) symbols of the module and verifies the CRC values of the symbols to ensure compatibility. In memory isolation for OOT drivers, symbols such as __kmalloc are relocated to OOT driver-specific symbols such as oot___kmalloc. When the OOT driver is compiled, the __kmalloc symbol is used in the driver code and is included in the UND symbols. However, the __kmalloc symbol is actually relocated to the oot___kmalloc symbol. The inconsistency between the expected symbol (__kmalloc) and the actual relocated symbol (oot___kmalloc) causes the issue.

Feedback