Container Service for Kubernetes:Use eRDMA in ACK clusters

Procedure

Step 1: Create a node pool

To enable eRDMA for nodes, you must use a separate node pool.

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side navigation pane, choose Nodes > Node Pools.

3. In the upper-right corner of the Node Pools page, click Create Node Pool.

   The following table describes the key parameters. For more information about other parameters, see Create a node pool.

   Parameter	Description
   Operating System	Select Alibaba Cloud Linux 3.x.
   Expected Nodes	Set the value to 0. You can add nodes after the node pool is created.
   CPU Policy (recommended)	Click Show Advanced Options and select Static.

Step 2: Add eRDMA-enhanced ECS instances

Configure eRDMA-enhanced ECS instances

The quota of network cards supported by an ECS instance varies based on the ECS instance type. Therefore, the ECS instance type affects the eRDMA quota, bandwidth, and configuration procedure.

You can refer to Overview of instance families to view the network card quotas of different instance types. Most instance types support only one network card. Perform the following steps to configure an eRDMA-enhanced ECS instance.

Add the eRDMA-enhanced ECS instance to the node pool

1. Add the ECS instance to the node pool.

   1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

   2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, choose Nodes > Node Pools.

   3. On the Node Pools page, click More in the Actions column of the node pool and click Add Existing Node.

   4. On the Select Existing ECS Instance wizard page, manually add an ECS instance. For more information, see Add existing ECS instances to an ACK cluster.

2. Install the latest eRDMA driver on each ECS instance.

Step 3: Run an application

1. Modify the Dockerfile to install the eRDMA repository and rebuild the container image so that the application created from the container image can use eRDMA. To do this, run the following commands:

   # Debian or Ubuntu: Make sure that the OS name and version in sources.list are the same as those you use. 
   wget -qO - https://mirrors.aliyun.com/erdma/GPGKEY | apt-key add - && echo "deb [ arch=amd64 ] https://mirrors.aliyun.com/erdma/apt/{OS|ubuntu} {Version|focal}/erdma main" | tee /etc/apt/sources.list.d/erdma.list && apt update && apt install -y libibverbs1 ibverbs-providers ibverbs-utils librdmacm1
   
   # Alibaba Cloud Linux or rhel: Find the yum.repos.d directory and create a file named erdma.repo. 
   cat > /etc/yum.repos.d/erdma.repo <<EOF
   [erdma]
   name = ERDMA Repository
   baseurl = http://mirrors.aliyun.com/erdma/yum/redhat/7/erdma/x86_64/
   gpgcheck = 0
   enabled = 1
   EOF
   yum install -y ethtool groff-base hwdata libnl3 pciutils
   sudo yum install libibverbs rdma-core librdmacm libibverbs-utils -y --disablerepo=* --enablerepo="erdma"
   
                           

2. Set the mode in which the application pods run.

   Configure pod parameters for the application.

   • Privileged: Configure the pods of the application to run in privileged mode to recognize eRDMA devices. For more information, see Scenario 1 in What do I do if the "Failed to initialize NVML: Unknown Error" error message appears when I run a pod that requests GPU resources on Alibaba Cloud Linux 3?

   • HostNetwork: Configure the pods of the application to run in the host network. For more information, see Use the host network.

3. Use the following YAML template to deploy a TensorFlow application:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: netperf
     name: netperf
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: netperf
     template:
       metadata:
         labels:
           app: netperf
       spec:
         hostNetwork: true
         containers:
         - command:
           - bash
           - -c
           - sleep 360000
           image: registry-vpc.cn-beijing.aliyuncs.com/wangbs/tftest
           imagePullPolicy: Always
           name: netperf
           resources:
             limits:
               cpu: "10"
             requests:
               cpu: "10"
           securityContext:
             privileged: true

4. Run the application to test eRDMA.

   1. Run the following command and record the IP addresses of the pods:

      kubectl get pod -o wide

      Expected output:

   2. Create two processes named PS and Worker. Specify the pod IP addresses in the following commands.

      PS:Worker:The following output is returned. You can verify the computing performance of the TensorFlow application.

      # Worker
      CUDA_VISIBLE_DEVICES= python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
        --ps_hosts=172.18.XX.XX:9100\
        --worker_hosts=172.18.XX.XX:9200 \
        --job_name=worker \
        --task_index=0 \
        --server_protocol=grpc+verbs \
        --variable_update=parameter_server \
        --local_parameter_device=cpu \
        --model=resnet50 \
        --device=cpu \
        --batch_size=16 \
        --data_format=NHWC
      
      # PS
      CUDA_VISIBLE_DEVICES= python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
        --ps_hosts=172.18.XX.XX:9100\
        --worker_hosts=172.18.XX.XX:9200 \
        --job_name=ps \
        --local_parameter_device=cpu \
        --task_index=0 \
        --device=cpu \
        --data_format=NHWC \
        --server_protocol=grpc+verbs