Elastic Compute Service:Configure eRDMA on an enterprise-level instance

Limits

Configure eRDMA on an enterprise-level ECS instance

Test the eRDMA write latency

You can install Perftest and test the write latency by using ib_write_lat on two enterprise-level instances that have eRDMA configured. For information about Perftest tests, see the Perftest test set section of the "Use eRDMA" topic.

Prepare the environment

1. Create two enterprise-level ECS instances that function as the server and client. Make sure that the eRDMA software stack is installed on the ECS instances and the ERIs are bound to the instances.

2. Make sure that the instances have valid network configurations and can communicate with each other over the internal network. For more information, see Connect ECS instances through an internal network.

Procedure

1. Connect to the two ECS instances.

   For more information, see Use Workbench to connect to a Linux instance over SSH.

2. Verify and confirm that the eRDMA configurations on both instances are correct.

   For more information, see Verify the correctness of eRDMA configurations of the "Use eRDMA" topic.

3. Install Perftest on each ECS instance.

   You can download the perftest package from the official perftest repository and install perftest, or use a Yellowdog Updater, Modified (YUM) or Advanced Packaging Tool (APT) repository to install perftest.

   Official perftest repository

   1. Enable public bandwidth for an ECS instance on which you want to install perftest. For more information, see Enable public bandwidth.

   2. Download the perftest package from the official perftest repository and install perftest.

   YUM or APT repository

   Note

   Different versions of perftest are included in the repositories of different Linux distributions. Incompatibility may occur. To prevent incompatibility, we recommend that you identify the Linux distribution run by the ECS instance on which you want to install perftest and install the perftest version included in the repository of the same Linux distribution. Otherwise, download the perftest package from the official perftest repository and install perftest.

   • Alibaba Cloud Linux 3/CentOS/Anolis OS

     sudo yum install perftest -y

   • Ubuntu

     sudo apt install perftest -y

4. Test whether the eRDMA network latency meets the expected performance.

   1. On the server-side instance, run the following command to start ib_write_lat as a server that listens for connections from the client:

      ib_write_lat -R -a -F

      • -R: uses RDMA Connection Manager (CM) to establish a connection.

        Important

        • By default, CPU-based instance families that support eRDMA install the eRDMA kernel-mode driver in Standard mode. In this mode, only the RDMA_CM connection establishement method is supported. For more information, see the Connection establishment method section of the "eRDMA" topic.

        • By default, Perftest establishes out of bandwidth (OOB) connections. When you perform Perftest tests on a CPU-based instance, you can specify the -R parameter on both the server and client to establish a connection by using the RDMA_CM method. Otherwise, an exception may occur during the connection establishment process.

        • You can also use the CLI method to enable compatibility between the RDMA_CM and OOB connection establishement methods. For more information, see Change the connection establishment mode of eRDMA to be compatible with bRPC. After you enable compatibility between the RDMA_CM and OOB connection establishement methods by using the CLI method, do not specify the -R parameter in the connection establishement command.

      • -a: sends test messages of all sizes. The size range is 2 bytes to 2^23 bytes. This allows you to test the impacts of different message sizes on latency.

      • -F: forcefully replaces an existing connection. If you do not require an existing RDMA connection, configure the -F parameter to ignore the existing connection and establish a new connection.

   2. On the client-side instance, run the following command to start ib_write_lat and connect to the server:

      ib_write_lat -R -a -F <server_ip>

      Replace <server_ip> with the private IP address of the ERI bound to the server-side ECS instance. For information about how to query IP addresses, see View IP addresses.

   3. Check the test results.

      After the client is tested, ib_write_lat outputs the test configuration information, connection information, and performance test results. The statistics include the minimum, maximum, and average latency.

      

      Latency data in the ib_write_lat test results

      • #bytes: the size of the payload of a test message. Valid values: 2 to 8388608. Unit: bytes. Different message sizes help you understand the performance under different loads.

      • #iterations: the number of iterations, which specifies the number of times messages of each size are repeatedly tested. A larger value indicates more stable statistics results, including average values.

      • t_min[usec]: the minimum latency recorded in all tests. Unit: microseconds. This value provides a reference for the best-case network latency.

      • t_max[usec]: the maximum latency recorded in all tests. Unit: microseconds. A large value may indicate specific network issues or transient traffic congestion.

      • t_typical[usec]: the typical latency recorded in tests. Unit: microseconds. In most cases, the value is the median of all tests.

      • t_avg[usec]: the average latency of all tests. Unit: microseconds. The average latency reflects the overall user experience on network latency.

      • t_stdev[usec]: the standard deviation of the latency. Unit: microseconds. A smaller value indicates more stable latency. A larger value indicates that the latency fluctuates.

      • 99% percentile[usec]: the latency value at the 99th percentile, which indicates that 99% of the latency values in the test results are lower than this value. Unit: microseconds. The data points at the 99th percentile help you understand latency performance in extreme cases.

      • 99.9% percentile[usec]: the latency value at the 99.9th percentile, which indicates that 99.9% of the latency values in the test results are lower than this value. Unit: microseconds. The data points at the 99.9th percentile help you understand latency performance in extreme cases.

      The preceding latency statistics provide you with a comprehensive understanding of the RDMA network performance to help you optimize network performance and troubleshoot network issues. For example, if the test results indicate a sudden increase in latency when test messages of a specific size are sent, you can check whether the network configurations or hardware performance meets your business requirements. If the test results indicate fluctuations in latency, you can check for traffic congestion or network instability issues.