Server Load Balancer:Stress testing methods for CLB

Stress testing methods

Take note of the following considerations when you perform a stress test on a CLB instance:

• Use short-lived connections if you want to test the forwarding capacity of a CLB instance.

  Typically, a CLB stress test is used to benchmark the forwarding capacity of a CLB instance, in addition to its session persistence and load balancing capabilities. Short-lived connections are suitable for benchmarking the capacities of both the CLB instance and its backend servers. You must make sure that the NLB instance has sufficient frontend ports to connect to clients if you use short-lived connections in a stress test.

• Use persistent connections if you want to test the throughput of the CLB instance. Persistent connections are suitable for testing the bandwidth limit or specific services.

  Set the timeout period on the stress testing tool to a small value, such as 5 seconds. If you set the timeout period to a large value, the average round-trip time (RTT) shown in the testing result may increase. As a result, it is difficult to judge whether the CLB instance is under extreme stress conditions. If you set the timeout period to a small value, you can judge whether the CLB instance is able to withstand the load based on the request success rate shown in the testing result.

• Host a static page on the backend server for stress testing. This can minimize the impacts of the application logic on the stress testing result.

• We also recommend that you use the following listener configurations:

  • Disable session persistence in case the CLB instance distributes network traffic only to some backend servers.

  • We recommend that you disable health checks to reduce the number of requests sent to backend servers.

  • Associate at least five elastic IP addresses (EIPs) with the NLB instance if the NLB instance supports up to 5,000 concurrent connections.

Stress testing tools

We recommend that you do not use Apache ab to perform stress tests. In high concurrency scenarios, the waiting time of Apache ab increases by increments of 3 seconds, such as 3 seconds, 6 seconds, and 9 seconds. Apache ab determines whether requests are successful based on the specified content length. If the NLB instance that you want to benchmark is associated with multiple backend servers, the actual length of the response content returned by these backend servers may be different from the specified content length. This makes the stress testing results inaccurate.

Possible causes of low scores on stress tests

Possible causes of low scores on stress tests:

• Insufficient client ports

  During a stress test, clients fail to establish connections with the NLB instance if the NLB instance does not have sufficient frontend ports. The NLB instance removes the timestamp property of TCP connections by default. As a result, the tw_reuse flag in the Linux stack becomes invalid. The tw_reuse flag is used to reuse connections that are in the time_wait state. Therefore, if this flag becomes invalid, connections in the time_wait state will accumulate and occupy the frontend ports of the NLB instance.

  Solution: Use persistent connections instead of short-lived connections on clients. In addition, use Reset (RST) packets to close connections by setting the SO_LINGER socket option.

• Full accept queues on backend servers

  If the accept queue on the backend servers is full, the backend servers can no longer return syn_ack packets. As a result, requests from clients time out.

  Solution: Run the sysctl -w net.core.somaxconn=1024 command to change the value of net.core.somaxconn and restart the application on the backend server. The default value of net.core.somaxconn is 128.

• The dependency of the application on the backend server becomes the performance bottleneck

  The traffic loads on the backend server are below the performance limit of the backend server. However, the application on the backend server may depend on another application, such as a database. Therefore, the dependency may also limit the performance of the NLB instance in stress tests.

  Solution: Clear applications that are no longer in use on the backend servers.

• Abnormal health status of backend servers

  If a backend server is declared unhealthy or the health status of the backend server changes frequently, the performance of the NLB instance in stress tests may be degraded.

  Solution: Disable health checks to reduce the number of requests sent to the backend servers.