Elastic Compute Service:Enable eRDMA on an enterprise-level instance

Test the eRDMA network latency against the expected performance.

1. On the server instance, run the following command to start ib_write_lat as a server that listens for connections from the client.

   ib_write_lat -R -a -F

   • -R: uses RDMA_CM to establish a connection.

     Important

     • By default, CPU-based instance types that support eRDMA install the eRDMA kernel driver in Standard mode. In this mode, only the RDMA_CM connection establishment method is supported. For more information, see Connection establishment methods.

     • By default, perftest establishes out-of-band (OOB) connections. When you run a perftest test on a CPU-based instance, you can specify the -R parameter on both the server and the client to use the RDMA_CM connection establishment method. Otherwise, an exception may occur when a connection is established.

     • You can also use the command line to make the RDMA_CM and OOB connection establishment methods compatible. For more information, see Modify the connection establishment modes of eRDMA and bRPC to ensure compatibility. After you make the methods compatible, you do not need to add the -R parameter to the command.

   • -a: runs tests for all message sizes, from 2 bytes to 2^23 bytes. This lets you test the effect of different message sizes on latency.

   • -F: forcefully overwrites any existing connection. This means that if a connection was previously established, using the -F option ignores the existing connection and forcefully establishes a new one.

2. On the client instance, run the following command to start ib_write_lat and connect to the server.

   ib_write_lat -R -a -F <server_ip>

   Replace <server_ip> with the private IP address of the network interface card (NIC) for which the ERI feature is enabled on the server ECS instance. For information about how to obtain an IP address, see View IP addresses.

3. View the test results.

   After the client test is complete, ib_write_lat outputs the test configuration information, connection information, and performance test results. The results include latency-related statistics, such as the minimum, maximum, and average latency.

   

   Description of latency data in the ib_write_lat test results

   • #bytes: The message size. This is the size of the payload used in the test, ranging from 2 bytes to 8,388,608 bytes. Different message sizes help you understand the performance under different loads.

   • #iterations: The number of iterations. This indicates how many times the test for each message size was repeated. A high number of iterations provides more stable averages and statistics.

   • t_min[usec]: The minimum latency. This is the minimum latency recorded in all measurements, in microseconds. This value provides a reference for the best-case network latency.

   • t_max[usec]: The maximum latency. This is the maximum latency recorded in all measurements, in microseconds. A high maximum latency may indicate network problems or transient congestion.

   • t_typical[usec]: The typical latency. This is the common latency in the test, in microseconds. It is usually the median of all measured values.

   • t_avg[usec]: The average latency. This is the average latency of all measured values, in microseconds. This value provides an overall impression of the network latency.

   • t_stdev[usec]: The standard deviation of the latency. This indicates the degree of variation in latency values, in microseconds. A smaller standard deviation means that the latency is more stable, while a larger standard deviation means that the latency fluctuates more.

   • 99% percentile[usec]: The 99th percentile of latency. This indicates that 99% of the measured values are below this value, in microseconds. This data point helps you understand latency performance in extreme cases.

   • 99.9% percentile[usec]: The 99.9th percentile of latency. This indicates that 99.9% of the measured values are below this value, in microseconds. This data point helps you understand latency performance in extreme cases.

   By combining this data, you can obtain a comprehensive understanding of the RDMA network performance and use it for network optimization and troubleshooting. For example, if you find that the latency suddenly increases at a specific message size, you may need to check whether the network configuration or hardware performance meets the requirements. If you see large fluctuations in latency, you may need to further investigate congestion or instability in the network.