Use eRDMA in Docker containers to allow applications to directly access physical devices to improve data transmission and communication efficiency - Elastic Compute Service

Elastic Remote Direct Memory Access (eRDMA) is a high-performance networking technology that can be used in Docker environments to allow container applications to bypass the kernel and directly access physical eRDMA devices on hosts. eRDMA helps improve data transfer speeds and communication efficiency and is suitable for scenarios in which large-scale data transfers and high-performance network communications need to be performed in containers. This topic describes how to configure eRDMA in Docker.

How eRDMA works

When you use eRDMA in a Docker environment, configure the --device option in Docker to map the following character devices to the container: /dev/infiniband/rdma_cm and /dev/infiniband/uverbsX. This allows user-space programs in the container to directly access eRDMA devices for data transfers without the need for kernel intervention. Take note of the following parameters in the preceding command:

/dev/infiniband/rdma_cm: is used to manage eRDMA connections. User-space programs can use this character device to establish, destroy, and manage connections to eRDMA devices and send or receive connection events.
/dev/infiniband/uverbsX: is used for user-space eRDMA operations. User-space programs can use this character device to communicate with eRDMA devices. The operations that user-space programs can perform by using the character device include starting eRDMA devices, creating and destroying eRDMA communication endpoints, and registering and deregistering memory buffers.
Note
X in /dev/infiniband/uverbsX specifies the index number of the device. The value of X may vary based on the operating system and configurations. You can run the ls /dev/infiniband | grep uverbs command to query the name of the character device.

Limits

You can use eRDMA only in the following Docker images:

Alibaba Cloud Linux 3 and Ubuntu 22.04 images, supported by Arm-based and x86-based instances
Alibaba Cloud Linux 2, CentOS 7, CentOS 8, Ubuntu 18.04, and Ubuntu 20.04 images, supported only by x86-based instances

Note

Anolis OS images are not supported.

Step 1: Deploy and configure Docker

Check whether eRDMA interfaces (ERIs) are deployed and work as expected on your instance and whether required drivers are installed on the instance.
For information about the deployment process, see Configure eRDMA on an enterprise-level instance.
Install Docker on your instance and download a Docker image. The installation and download operations vary based on the image type.
- If the Docker image is an Alibaba Cloud Linux or CentOS image, see Deploy and use Docker on ECS instances to learn how to install Docker and download the image.
- If the Docker image is a Ubuntu image, see Install Docker Engine on Ubuntu to learn how to install Docker and download the image.
Run the following command to expose the character devices that are used for eRDMA to the container:
```
sudo docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --ulimit memlock=-1 -t -i <IMAGE ID> /bin/bash
```
Take note of the following parameters in the preceding command:
- --net=host: places the container in host network mode. Application programs in a container that runs in host network mode have direct access to the network interfaces and network configurations of the host and provide the same network communication capabilities as the host.
- --device=/dev/infiniband/uverbs0 and --device=/dev/infiniband/rdma_cm: expose user-space character devices that are used for eRDMA to the container.
- --ulimit memlock=-1: sets memlock (max locked-in-memory address space) to unlimited, which specifies that the maximum amount of memory that non-root users can lock is unlimited. This ensures that non-root users can lock the amount of memory that is required by eRDMA applications to implement the eRDMA functionality.
- <IMAGE ID>: specifies the ID of the Docker image. Configure this parameter based on the actual environment. You can run the sudo docker images command to query the IDs of Docker images.

Step 2: Install the eRDMA driver package in Docker

Before you use eRDMA, install the eRDMA user-space driver package in the container. Alibaba Cloud provides Yellowdog Updater Modified (YUM) and APT repositories to help you deploy required packages.

Important

In the following examples, URLs that start with http://mirrors.cloud.aliyuncs.com are the internal URLs of the repositories. If you want to access the repositories over the Internet, replace http://mirrors.cloud.aliyuncs.com with http://mirrors.aliyun.com to obtain the public URLs. Data is transferred to or from the Internet when you access the repositories over the Internet. You may be charged data transfer fees. For information about how data transfers to or from the Internet are billed, see Public bandwidth.

CentOS 7 or CentOS 8

Run the following command in the container to create an erdma.repo file in the /etc/yum.repos.d directory:
```
sudo vim /etc/yum.repos.d/erdma.repo
```

Add the following content to the erdma.repo file, save the file, and then exit:

[erdma]
name = ERDMA Repository
baseurl = http://mirrors.cloud.aliyuncs.com/erdma/yum/redhat/$releasever/erdma/$basearch/
gpgcheck = 1
enabled = 1
gpgkey = http://mirrors.cloud.aliyuncs.com/erdma/GPGKEY

Run the following command to update the YUM cache:
```
sudo yum makecache
```

Run the following command to install the user-space driver package:

sudo yum install libibverbs rdma-core librdmacm libibverbs-utils -y

Alibaba Cloud Linux

Run the following commands in the container to add a YUM repository:
```
sudo yum-config-manager \
 --add-repo \
 http://mirrors.cloud.aliyuncs.com/erdma/yum/alinux/erdma.repo
```
Note
If the yum-config-manager utility is not installed in your container, run the sudo yum install -y yum-utils command to install the yum-utils package. The yum-config-manager utility is included in the yum-utils package and used to manage YUM configurations.
Run the following command to update the YUM cache:
```
sudo yum makecache
```

Run the following command to install the user-space driver package:

sudo yum install libibverbs rdma-core librdmacm libibverbs-utils -y

(Optional) Run the following command to install Network Accelerator (NetACC).
NetACC is a user-space network acceleration library that leverages the benefits of eRDMA, such as low latency, high throughput, kernel bypass, and protocol stack offload, to be compatible with socket interfaces and accelerate existing TCP applications. For more information, see NetACC.
```
sudo yum install netacc -y
```

Ubuntu 18.04, Ubuntu 20.04, or Ubuntu 22.04

Run one of the following commands in the container based on the operating system version to add a GNU Privacy Guard (GPG) key.

Ubuntu 18.04 or Ubuntu 20.04

wget -qO - http://mirrors.cloud.aliyuncs.com/erdma/GPGKEY | sudo apt-key add -

Ubuntu 22.04

wget -qO - http://mirrors.cloud.aliyuncs.com/erdma/GPGKEY | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/erdma.gpg

Run one of the following commands based on the operating system version to add an APT repository.

Ubuntu 20.04

echo "deb [ arch=amd64 ] http://mirrors.cloud.aliyuncs.com/erdma/apt/ubuntu focal/erdma main" | sudo tee /etc/apt/sources.list.d/erdma.list

Ubuntu 18.04

echo "deb [ arch=amd64 ] http://mirrors.cloud.aliyuncs.com/erdma/apt/ubuntu bionic/erdma main" | sudo tee /etc/apt/sources.list.d/erdma.list

Ubuntu 22.04

echo "deb [ ] http://mirrors.cloud.aliyuncs.com/erdma/apt/ubuntu jammy/erdma main" | sudo tee /etc/apt/sources.list.d/erdma.list

Run the following command to update the APT repository:
```
sudo apt update
```

Run the following command to install the user-space driver:

sudo apt install libibverbs1 ibverbs-providers ibverbs-utils librdmacm1