Problem description
During stress testing with Performance Testing Service (PTS), you may see errors or timeouts even though your backend servers show low CPU, memory, and I/O utilization. This typically means the bottleneck is in the network access layer between PTS and your backend, not in your application servers.
Cause
Requests from PTS pass through one or more intermediate services before reaching the backend. These services include load balancers, firewalls, DDoS protection, and CDN nodes. Each service has its own capacity limits and security policies. When a limit is reached, requests are dropped or rejected before they reach your backend. As a result, backend metrics stay low while errors spike.
Quick diagnosis
Use the following table to identify which network access layer is causing errors:
| Symptom | Likely cause | Section |
|---|---|---|
| HTTP 503 errors; backend has no traffic or logs | SLB throttling | SLB throttling |
| Sudden error spike mid-test; security logs show blocked requests | Anti-DDoS or WAF blocking | Anti-DDoS or WAF blocking |
| Errors when traffic approaches or exceeds historical peak | CDN or ESA capacity limits | CDN or ESA capacity |
Solution
Identify which network access layer product sits in front of your backend, then follow the matching section.
Server Load Balancer (SLB) throttling
Symptoms
-
HTTP 503 errors during stress testing
-
Backend servers show no corresponding traffic or log entries
-
Errors appear under HTTPS workloads or when layer 7 session persistence is enabled
Cause
SLB instances enforce per-IP throttling limits. When SLB is the outermost service layer and the API uses HTTPS or layer 7 session persistence, SLB processing overhead increases. The single IP address of the SLB instance may hit its throttling threshold, causing 503 errors instead of forwarding requests to the backend.
Resolution
-
Check the specification limits of your SLB instance:
Metric Description Maximum connections Total concurrent connections the instance supports Connections per second (CPS) New connections established per second Queries per second (QPS) HTTP/HTTPS requests processed per second -
Verify that the bandwidth allocation matches your stress testing volume based on your purchased billing type.
-
If the SLB specification is the bottleneck, upgrade to a higher-spec instance before re-running the test.
For details on 503 errors from SLB, see 503 (Service Unavailable).
Anti-DDoS or WAF blocking test traffic
Symptoms
-
Sudden spike in errors partway through a stress test
-
Anti-DDoS or Web Application Firewall (WAF) logs show blocked requests
-
Stress testing results are inconsistent, or the test fails entirely
Cause
Stress testing traffic patterns -- high request rates from a limited set of IPs -- closely resemble Challenge Collapsar (CC) and DDoS attacks. Anti-DDoS and WAF protection policies detect these patterns and block the traffic.
Resolution
-
Check Anti-DDoS or WAF logs to confirm that the blocked traffic is legitimate PTS stress testing traffic.
-
Evaluate the impact on your production environment.
-
Temporarily disable the relevant protection policies, or add PTS stress testing traffic to the allowlist.
-
Re-enable the protection policies after the stress test completes.
For more information, see:
CDN or ESA capacity limits
Symptoms
-
Errors or degraded performance when stress testing traffic approaches or exceeds your historical peak
-
Content Delivery Network (CDN) or Edge Security Acceleration (ESA) points of presence (POPs) return errors
Cause
CDN and ESA POPs allocate capacity based on your historical traffic patterns. A stress test that surpasses your existing peak may exceed the allocated capacity at specific POPs.
Resolution
If your stress test is expected to approach or exceed your current peak traffic level, submit a ticket before running the test to report the target CDN POPs and ESA POPs.