Container Service for Kubernetes:Use eRDMA to accelerate container networking in ACK clusters

Step 1: Install ACK eRDMA Controller

You can perform the following steps to install ACK eRDMA Controller.

1. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Operations > Add-ons.

2. On the Add-ons page, click the Networking tab, find ACK eRDMA Controller, follow the instructions on the page to configure and install the component.

   Parameter	Description
   preferDriver Driver type	Select the type of the eRDMA driver used on the cluster nodes. Valid values: default: The default driver mode. compat: The driver mode that is compatible with RDMA over Converged Ethernet (RoCE). ofed: The ofed-based driver mode, which is applicable to GPU models. For more information about the types of drivers, see Use eRDMA.
   Specifies whether to assign all eRDMA devices of nodes to pods	Valid values: True: If you select this check box, all eRDMA devices on the node are assigned to the pod. False: If you do not select this check box, the pod is assigned an eRDMA device based on the non-uniform memory access (NUMA) topology. You must enable the static CPU policy for the node to ensure that NUMA can be allocated to pods and devices. For more information about how to configure CPU policies, see Create a node pool.

   In the left-side navigation pane, choose Workloads > Pods. On the Pods page, select the ack-erdma-controller namespace to view the status of pods and ensure that the component runs as expected.

Step 2: Use eRDMA to accelerate container networking

After you install ACK eRDMA Controller, you can use the following configuration to enable eRDMA for the pod.

spec:
  containers:
  - name: erdma-container
    resources:
      limits:
        aliyun/erdma: 1

/# ls /dev/infiniband/
rdma_cm  uverbs0

metadata:
  annotations:
    network.alibabacloud.com/erdma-smcr: "true"

Scenario 1: GPU models use eRDMA to accelerate NCCL

1. When you install ACK eRDMA Controller based on Step 1: Install ACK eRDMA Controller, set the preferDriver parameter to ofed to accelerate Nvidia Collective Communication Library (NCCL).

2. Add GPU-accelerated nodes to the node pool. For more information, see Create a node pool.

3. Install the eRDMA-related packages when you build an application container image.

   View the installed eRDMA-related packages

   # Debian or Ubuntu: Make sure that the OS name and version in sources.list are the same as those you use. 
   wget -qO - https://mirrors.aliyun.com/erdma/GPGKEY | apt-key add - && echo "deb [ arch=amd64 ] https://mirrors.aliyun.com/erdma/apt/{OS|ubuntu} {Version|focal}/erdma main" | tee /etc/apt/sources.list.d/erdma.list && apt update && apt install -y libibverbs1 ibverbs-providers ibverbs-utils librdmacm1
   
   # For Alibaba Cloud Linux or Red Hat Enterprise Linux (RHEL), specify the OS repository address in the yum.repos.d directory. 
   cat > /etc/yum.repos.d/erdma.repo <<EOF
   [erdma]
   name = ERDMA Repository
   baseurl = http://mirrors.aliyun.com/erdma/yum/redhat/7/erdma/x86_64/
   gpgcheck = 0
   enabled = 1
   EOF
   yum install --disablerepo=*  --enablerepo erdma -y libibverbs ibverbs-providers ibverbs-utils librdmacm

4. Run a GPU application that uses eRDMA in a cluster. nccl-test is used as an example.

   View the sample template of a GPU application that uses eRDMA

   apiVersion: apps/v1
   kind: StatefulSet
   metadata:
     name: nccltest
   spec:
     selector:
       matchLabels:
         app: nccltest
     serviceName: "nccltest"
     replicas: 2
     template:
       metadata:
         labels:
           app: nccltest
       spec:
         hostNetwork: true 
         dnsPolicy: ClusterFirstWithHostNet
         containers:
         - env:
           - name: NCCL_SOCKET_IFNAME
             value: "eth0"
           - name: NCCL_DEBUG
             value: "INFO"
           - name: NCCL_IB_GID_INDEX
             value: "1"
           image: <nccl-test-image-with-erdma>
           imagePullPolicy: Always
           name: nccltest
           securityContext:
             privileged: true
           resources:
             limits:
               nvidia.com/gpu: "8"
               aliyun/erdma: "1"
             requests:
               nvidia.com/gpu: "8"
               aliyun/erdma: "1"

5. Verify that eRDMA is used by NCCL.

   You can check the communication type and the number of network interfaces used by NCCL in the application logs. Example:

   

   The command output indicates that the erdma_0 and erdma_1 eRDMA devices is accelerated.