Elastic Compute Service:Use eRDMA in Docker containers

Step 1: Configure eRDMA for an instance

To make sure that the eRDMA device works as expected on an Elastic Compute Service (ECS) instance on which you want to deploy Docker, configure eRDMA on the instance by performing the following operations: Check that the instance type of the instance supports eRDMA, bind eRDMA-enabled elastic network interfaces (ENIs), also known as eRDMA interfaces (ERIs), to the instance, and then install the eRDMA driver on the instance.

• For information about how to configure eRDMA on an enterprise-level CPU-based instance, see Configure eRDMA on an enterprise-level instance.

(Optional) Step 2: Deploy Docker on the ECS instance

To use Docker on your instance, you must install Docker on the instance.

Perform the following steps:

1. Connect to an ECS instance.

   For more information, see Use Workbench to log on to a Linux instance over SSH.

2. Run the following command to check whether the Docker is deployed on the instance:

   sudo docker -v

   If a Docker version is returned, Docker is deployed as expected, as shown in the following figure.

   

   The following command output indicates that Docker is not deployed or does not work as expected. Go to the next step to deploy Docker.

   

3. Deploy Docker on the instance. The operations vary based on the operating system.

   • Alibaba Cloud Linux or CentOS: See Install Docker.

   • Ubuntu: See Install Docker Engine on Ubuntu.

(Optional) Step 3: Create a Docker image

If no images are deployed in your Docker container, you can create an image based on your actual application requirements.

In this example, Alibaba Cloud Linux base images are downloaded in Docker, which requires Internet connectivity.

Perform the following steps:

1. Connect to an ECS instance where a Docker instance resides.

   For more information, see Use Workbench to connect to a Linux instance over SSH.

2. Run the following command to start the Docker service:

   sudo systemctl start docker

3. Run the following command to download the Alibaba Cloud Linux Docker image:

   sudo docker pull alibaba-cloud-linux-<image_version>-registry.<region_ID>.cr.aliyuncs.com/alinux<image_version>/alinux<image_version><:TAG>

   In the preceding command, configure the following parameters:

   • <image_version>: the Alibaba Cloud Linux version. Example: 2 or 3.

   • <region_ID>: the region ID of the Docker image. Example: cn-hangzhou.

   • <:TAG>: optional. The tag of the Docker image. If you configure this parameter, the specified Docker image version is downloaded. If you leave this parameter empty, the latest Docker image version is downloaded.

   View the region and version information of an image

   1. Log on to the Container Registry console.

   2. In the left-side navigation pane, click Artifact Center.

   3. To view the information of the Alibaba Cloud Linux 2 Docker image, click alinux2/alinux2. To view the information of the Alibaba Cloud Linux 3 Docker image, click alinux3/alinux3.

      For example, the following figure shows the information of the Alibaba Cloud Linux 3 Docker image. The ① section indicates the region in which the Docker image resides, and the ② section indicates the version information of the Docker image.

      

   Sample commands:

   • Sample command used to download version 220901.1 of the Alibaba Cloud Linux 3 Docker image in the China (Hangzhou) region:

     sudo docker pull alibaba-cloud-linux-3-registry.cn-hangzhou.cr.aliyuncs.com/alinux3/alinux3:220901.1

   • Sample command used to download the latest version of the Alibaba Cloud Linux 2 Docker image in the China (Hangzhou) region:

     sudo docker pull alibaba-cloud-linux-2-registry.cn-hangzhou.cr.aliyuncs.com/alinux2/alinux2

4. Run the following command to check whether the images are downloaded:

   sudo docker images

   The command output indicates that version 220901.1 of the Alibaba Cloud Linux 3 Docker image and the latest version of the Alibaba Cloud Linux 2 Docker image are downloaded.

   

Step 4: Start the container and mount the eRDMA device

When you use eRDMA in a Docker container, you can map (mount) the following eRDMA character devices to the container: /dev/infiniband/rdma_cm and dev/infiniband/uverbsX. This allows user-space programs in the container to bypass the kernel and directly access eRDMA devices for data transfers. To map the preceding character devices to the container, configure the --device parameter in the docker run command used to run the container. For more information, see the How eRDMA works in a Docker container section of this topic.

Perform the following steps:

1. Connect to the Linux ECS instance.

   For more information, see Use Workbench to log on to a Linux instance over SSH.

2. Run the following command to start a Docker container and map the eRDMA character devices to the container:

   sudo docker run --net=host --device=/dev/infiniband/uverbsX --device=/dev/infiniband/rdma_cm --ulimit memlock=-1 -t -i <IMAGE ID> /bin/bash

   Take note of the following items:

   • --net=host: places the container in host network mode. Application programs in a container that runs in host network mode have direct access to the network interfaces and network configurations of the host and provide the same network communication capabilities as the host.

   • --device=/dev/infiniband/uverbsX and --device=/dev/infiniband/rdma_cm: maps user-space character devices that are used for eRDMA to the container.

     To view the character device name, run the following command. X in uverbsX specifies the index of the uverbs character device.

     ls /dev/infiniband | grep uverbs

     

   • --ulimit memlock=-1: sets maxlockedmemory (max locked-in-memory address space) to unlimited, which specifies that the maximum amount of memory that non-root users can lock is unlimited. This ensures that non-root users can lock the amount of memory required by eRDMA applications to enable the eRDMA functionality.

   • <IMAGE ID>: specifies the ID of the Docker image. Replace this parameter with the actual Docker image ID. To query the IDs of Docker images, run the sudo docker images command.

     Run the following command to query the IDs of the images:

     sudo docker images

     In this example, the Alibaba Cloud Linux images deployed in Step 3 are used. The following command output is returned.

     

   Sample command

   sudo docker run --net=host --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --ulimit memlock=-1 -t -i e8d9a60b6967 /bin/bash

Step 5: Install the eRDMA driver in the Docker container

To use eRDMA in a container, you must install the user-space eRDMA driver package. This allows the container to recognize eRDMA device information. Alibaba Cloud provides Yellowdog Updater Modified (YUM) and Advanced Package Tool (APT) repositories to help you deploy required packages.

Perform the following steps:

1. Connect to the Linux ECS instance.

   For more information, see Use Workbench to log on to a Linux instance over SSH.

2. Access the container.

   When you run the command in Step 4, you already have access to the Docker container. Install the user-space eRDMA driver package in the container.

   1. Run the following command to query the container ID:

      sudo docker ps

      In this example, the container started in Step 4 is used. Sample command output:

      

   2. Run the following command to access the container:

      sudo docker exec -it <CONTAINER ID> /bin/bash

      Replace CONTAINER ID with the ID of the container you queried in the previous step.

3. After you access the container, install the user-space driver package in the container.

   Important

   In the following examples, URLs that start with http://mirrors.cloud.aliyuncs.com are the internal URLs of the repositories. If you want to access the repositories over the Internet, replace http://mirrors.cloud.aliyuncs.com with http://mirrors.aliyun.com to obtain the public URLs. Data is transferred to or from the Internet when you access the repositories over the Internet. You may be charged data transfer fees. For information about how data transfers to or from the Internet are billed, see Public bandwidth.

   CentOS 7 and CentOS 8

   1. Run the following command in the container to create an erdma.repo file in the /etc/yum.repos.d directory:

      sudo vim /etc/yum.repos.d/erdma.repo

   2. Add the following content to the erdma.repo file, save the file, and then exit:

      [erdma]
      name = ERDMA Repository
      baseurl = http://mirrors.cloud.aliyuncs.com/erdma/yum/redhat/$releasever/erdma/$basearch/
      gpgcheck = 1
      enabled = 1
      gpgkey = http://mirrors.cloud.aliyuncs.com/erdma/GPGKEY

   3. Run the following command to update the YUM cache:

      sudo yum makecache

   4. Run the following command to install the user-space driver package:

      sudo yum install libibverbs rdma-core librdmacm libibverbs-utils -y 

   Alibaba Cloud Linux

   1. Run the following commands in the container to add a YUM repository:

      sudo yum-config-manager \
       --add-repo \
       http://mirrors.cloud.aliyuncs.com/erdma/yum/alinux/erdma.repo

      Note

      If the yum-config-manager utility is not installed in your container, run the sudo yum install -y yum-utils command to install the yum-utils package. The yum-config-manager utility is included in the yum-utils package and used to manage YUM configurations.

   2. Run the following command to update the YUM cache:

      sudo yum makecache

   3. Run the following command to install the user-space driver package:

      sudo yum install libibverbs rdma-core librdmacm libibverbs-utils -y

   Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04

   1. Run one of the following commands in the container based on the operating system version to add a GNU Privacy Guard (GPG) key.

      • Ubuntu 18.04 and Ubuntu 20.04

        wget -qO - http://mirrors.cloud.aliyuncs.com/erdma/GPGKEY | sudo apt-key add -

      • Ubuntu 22.04

        wget -qO - http://mirrors.cloud.aliyuncs.com/erdma/GPGKEY | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/erdma.gpg

   2. Run one of the following commands based on the operating system version to add an APT repository.

      • Ubuntu 20.04

      echo "deb [ arch=amd64 ] http://mirrors.cloud.aliyuncs.com/erdma/apt/ubuntu focal/erdma main" | sudo tee /etc/apt/sources.list.d/erdma.list

      • Ubuntu 18.04

      echo "deb [ arch=amd64 ] http://mirrors.cloud.aliyuncs.com/erdma/apt/ubuntu bionic/erdma main" | sudo tee /etc/apt/sources.list.d/erdma.list

      • Ubuntu 22.04

      echo "deb [ ] http://mirrors.cloud.aliyuncs.com/erdma/apt/ubuntu jammy/erdma main" | sudo tee /etc/apt/sources.list.d/erdma.list

   3. Run the following command to update the APT repository:

      sudo apt update

   4. Run the following command to install the user-space driver package:

      sudo apt install libibverbs1 ibverbs-providers ibverbs-utils librdmacm1 -y

4. Run the following command to view eRDMA device information in the container:

   ibv_devinfo

   

   The preceding command output indicates that the eRDMA device can be recognized as expected.

After eRDMA is configured in the Docker container, you can use Shared Memory Communications over Remote Direct Memory Access (SMC-R) or Network Accelerator (NetACC) to integrate eRDMA into TCP applications in the container for application acceleration. For more information, see Overview of adapting eRDMA and applications.