how to replace UIO drivers with VFIO drivers - Elastic Compute Service

This topic describes how to replace a Userspace I/O (UIO) driver with a Virtual Function I/O (VFIO) driver to resolve the exceptions that occur in Data Plane Development Kit (DPDK) applications at runtime for an Elastic Compute Service (ECS) instance.

Prerequisites

Huge pages are configured for the instance. For more information, see How to adjust the huge page memory of Linux instances.
DPDK is installed on the instance. You can visit the Data Plane Development Kit (DPDK*) page for information about DPDK.

Background information

If DPDK applications are deployed on instances of sixth-generation instance types (such as g6, c6, and r6 instance types) or later, exceptions may occur when you run the applications. For example, if you use Pktgen-DPDK to test the packet forwarding rate of an instance, the igb_uio port to which a network interface controller (NIC) is bound may not be found and the following error message may be reported. To resolve the exceptions that occur in DPDK applications, you can replace UIO drivers with VFIO drivers.

EAL: eal_parse_sysfs_value(): cannot open sysfs value /sys/bus/pci/devices/0000:00:06.0/uio/uio0/portio/port0/start

Procedure

Connect to an regular ECS instance or an ECS bare metal instance.
For more information, see Connect to a Linux instance by using a password or key.
Check the configurations of the GRand Unified Bootloader (GRUB).
```
cat /proc/cmdline
```
- If the GRUB configurations include the intel_iommu=on parameter and do not include the iommu=pt parameter, proceed to Step 4.
- Otherwise, modify the GRUB configurations to make sure that the configurations include the intel_iommu=on parameter and do not include the iommu=pt parameter, and then proceed to Step 3.
  The following figure shows an example of GRUB configurations that do not include the intel_iommu=on parameter or iommu=pt parameter. In this case, you must add the intel_iommu=on parameter to the GRUB configurations.
Add the intel_iommu=on parameter to the /etc/default/grub GRUB configuration file.
1. Run the following command to open the GRUB configuration file:
```
sudo vim /etc/default/grub
```
2. Press the I key to enter Insert mode, add intel_iommu=on to the end of the line for GRUB_CMDLINE_LINUX, and then save the configuration file.
  The following figure shows an example of the modified configuration file.
3. Run the following command to apply the modified configurations:
```
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
```
4. Run the following command to restart the instance. Then, reconnect to the instance.
```
reboot
```
  Warning
  The restart operation stops the instance for a short period of time and may interrupt services that are running on the instance. We recommend that you restart the instance during off-peak hours.
Run the following commands to install VFIO and VFIO-PCI drivers:
```
sudo modprobe vfio && \
sudo modprobe vfio-pci
```

Run the following command to enable no-IOMMU mode:

sudo bash -c 'echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode'

Run the following command to view and record the bus-info value of an NIC that is bound to the instance:
```
ethtool -i ethX
```
Replace ethX with the ID of an NIC that is bound to the instance. In this example, eth1 is used.
Switch to the usertools directory under the installation directory of DPDK. Then, run the following command with the bus-info value of an NIC to bind the NIC to the VFIO-PCI driver:
```
sudo ./dpdk-devbind.py -b vfio-pci 0000:00:06.0
```
Note
In this example, the bus-info value of the NIC is 0000:00:06.0. In actual scenarios, replace it with the bus-info value of your NIC.
Run the ./dpdk-devbind.py --status command to view the binding status of NICs. The following command output indicates that the NIC whose bus-info value is 0000:00:06.0 is bound to the VFIO-PCI driver.
Switch to the build/app directory under the installation directory of DPDK and run one of the following commands based on the DPDK version to start DPDK.
- If the DPDK version is 18.02 or later, run the following command:
```
sudo ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 -ai
```
- If the DPDK version is earlier than 18.02, run the following command:
```
sudo ./testpmd -w 0000:00:06.0 -c 0x3 -- --total-num-mbufs=2048 --disable-hw-vlan -ai
```
Note
-w specifies the bus-info value of an NIC. In this example, -w is set to 0000:00:06.0. total-num-mbufs specifies the size of huge pages. In this example, total-num-mbufs is set to 2048. In actual scenarios, configure these parameters based on your business requirements.
The following command output indicates that DPDK is started.