Object Storage Service:Troubleshoot 5xx errors for CDN back-to-origin requests

Description

When you use Alibaba Cloud CDN to accelerate access to your website, client requests are routed through a multi-tier network: from a Tier-1 point of presence (POP) to a Tier-2 POP, and finally to your origin server. If Alibaba Cloud CDN cannot retrieve content from the origin server, you may encounter the following errors:

• 502 Bad Gateway: The origin server returned an invalid response, or SSL/TLS negotiation failed.

• 503 Service Temporarily Unavailable: The origin server is temporarily unable to handle the request due to overload or maintenance.

• 504 Gateway Time-out: The origin server did not respond within the expected time frame.

Causes

To identify the specific cause, examine the X-Swift-Error field in the HTTP response header. Common error messages include forward retry timeout or orig response 5xx error. You can use cURL or your browser's developer tools (Network tab) to inspect response headers and request duration.

The root cause typically falls into one of these categories:

• Configuration issues - If 5xx errors occur for all requests, the problem is usually a misconfiguration in Alibaba Cloud CDN or an issue with the origin server:

  • The origin server is inaccessible, or its domain name cannot be resolved.

  • Back-to-origin routing over HTTPS is configured in Alibaba Cloud CDN, but the origin server does not support HTTPS.

  • Server Name Indication (SNI) is enabled for the origin server but not in Alibaba Cloud CDN.

• Network or capacity issues - If errors occur intermittently or only in specific regions, the cause is likely related to:

  • Security policies on the origin server blocking CDN POP IP addresses.

  • Unstable network connections, inconsistent back-to-origin routes, or slow response times from dynamic interfaces.

  • Origin server bandwidth limits exceeded or insufficient server resources.

Solutions

Alibaba Cloud CDN caches content from origin servers on globally distributed POPs to accelerate access. When a resource is requested for the first time or when cached content expires, the request is forwarded to the origin server. The origin server's response time and time to first byte (TTFB) are critical to back-to-origin performance. Use the following troubleshooting methods based on your origin server configuration:

Scenario 1: Use a local host or ECS instance as the origin server

If your origin server is a local host or an Elastic Compute Service (ECS) instance, follow these steps:

Step 1: Obtain and analyze HTTP response headers

Use cURL or Wget to inspect HTTP response headers and identify error details:

curl -vo [$File_Name] [$IP]

If the X-Swift-Error response field contains "orig response 5xx error" or similar messages, the issue is with the origin server's response or timeout.

Step 2: Test network performance by affected region

Use third-party monitoring tools such as Networkbench or 17CE to check network performance across different regions:

• Single region affected: The error may be related to connectivity between a specific POP and your origin server. Run multiple network tests to verify if the issue persists. If the problem causes significant impact, document the response headers and contact Alibaba Cloud technical support.

• Multiple regions affected: Test connection quality using the following commands:

  time telnet [$Test_IP] [$Test_Port]
  ping [$Test_IP]

  Note

  • The telnet command measures connection establishment time.

  • The ping command tests network connectivity and latency.

  • [$Test_IP]: The IP address of the origin server or CDN POP to test.

  • [$Test_Port]: The port number (typically 80 for HTTP or 443 for HTTPS).

Step 3: Check bandwidth and QPS of your domain

Verify that your origin server has sufficient capacity to handle back-to-origin traffic:

1. Log on to the Alibaba Cloud CDN console.

2. In the left-side navigation pane, choose Content Delivery > Monitoring & Usage Analytics > Real-time Monitoring to check recent back-to-origin bandwidth and queries per second (QPS). Sudden spikes may cause origin server timeouts.

3. If you observe a back-to-origin bandwidth spike, investigate whether:

   • New resources were added to the origin server without proper caching configuration.

   • Cache rules are properly configured for frequently accessed resources.

   For applications with large volumes of gaming data or images, use the prefetch feature to proactively deploy resources to CDN POPs and configure longer cache time-to-live (TTL) values for frequently accessed resources.

4. Review cache configurations on both Alibaba Cloud CDN and your origin server. Note that no-cache response headers from the origin server take precedence over CDN cache settings. If no cache rules are configured, the default TTL ranges from 10 to 3,600 seconds. For more information, see What is caching?.

   If you are new to Alibaba Cloud CDN caching, configure a maximum TTL for directories and create separate policies for non-cacheable resources with priority-based matching. For more information, see Configure CDN cache expiration time.

Step 4: Verify origin server connectivity and DNS resolution

CDN POPs must be able to reach your origin server over the Internet. Verify that your origin server's IP address, port, and domain name are correctly configured and accessible:

1. Test origin server accessibility: Use the ping command to verify connectivity:

   ping [$IP]

   Note

   [$IP] is the IP address of your origin server.

2. Test port accessibility: If the origin server's port is inaccessible or returns 5xx errors, test using telnet. A Connection timed out error indicates port issues:

   1. For HTTP (port 80):

      telnet [$IP] 80

   2. For HTTPS (port 443) or custom ports, test the appropriate port number.

   3. Perform Layer 7 testing by mapping the origin server IP address in your local hosts file and using cURL:

      • For HTTP (port 80): curl -voa http://[$Domain] -x [$IP]:80

        Note

        [$Domain] is your accelerated domain name.

      • For HTTPS (port 443): curl -voa https://[$Domain] --resolve [$Domain]:443:[$IP]

3. Verify DNS resolution: If your origin server uses a domain name, ensure it resolves correctly. Test using ping or nslookup. If you encounter errors like unknown host or server can't find example.aliyundoc.com: NXDOMAIN, configure DNS resolution. For more information, see Configure a CNAME record.

Step 5: Review back-to-origin HTTPS configuration

SSL/TLS configuration mismatches between CDN and your origin server can cause 5xx errors. Verify the following:

1. Port 443 configuration: If back-to-origin routing uses port 443, ensure:

   • Port 443 is open on the origin server.

   • A valid SSL/TLS certificate is installed on the origin server.

   • If HTTPS is not supported, change the back-to-origin port to 80 or configure an SSL/TLS certificate.

2. Protocol-following configuration: If back-to-origin routing is configured to follow the protocol used by the client, and clients use HTTPS, ensure your origin server supports HTTPS. Otherwise, disable protocol-following or configure HTTP for back-to-origin routing.

3. Test HTTPS accessibility: Use cURL to verify:

   curl -voa https://[$Domain] --resolve [$Domain]:443:[$IP]

4. Alternatively, modify your local hosts file and test using a browser. If you see Your connection is not private, HTTPS is not properly configured on the origin server.

Scenario 2: Use OSS as the origin server

If you use Alibaba Cloud CDN with Object Storage Service (OSS) as the origin server, follow these steps:

1. Check for OSS traffic throttling: Use the --probe-item option in the ossutil tool to verify if your outbound bandwidth exceeds 10 Gbit/s or QPS exceeds 10,000. Exceeding these limits triggers traffic throttling, causing delayed request processing. For more information, see Limits and performance metrics.

   Note

   • For usage instructions, see View options.

   • Throttled requests are delayed, not rejected.

2. Verify origin server bandwidth capacity: Check whether back-to-origin traffic exceeds your origin server's bandwidth limit. When maximum bandwidth is reached, network and application layer issues may occur.

   Note

   • Use the iperf tool to measure traffic between POPs.

   • Use the netstat tool to check the number of active network connections.

3. Review firewall and security policies: Verify that firewall rules do not block CDN POP IP addresses or rate-limit back-to-origin requests.

4. Examine origin server logs: If previous steps do not reveal issues, check Web server error logs for 5xx errors. If no application-layer errors are found, test the connection from your origin server to CDN POPs to identify network-related issues.

   Note

   If your origin server frequently reaches maximum bandwidth or experiences packet loss, consider increasing the send_timeout or read_timeout parameter in your nginx.conf file.

Best practices

To minimize 5xx errors and improve CDN performance, implement the following best practices:

• Monitor origin server health: Set up proactive monitoring for origin server metrics including CPU, memory, bandwidth, and response times.

• Optimize cache configuration: Configure appropriate cache TTL values to reduce back-to-origin requests and minimize origin server load.

• Use prefetch for large content: Proactively deploy large files to CDN POPs during off-peak hours to prevent origin server overload.

• Test HTTPS configuration: Before enabling HTTPS for back-to-origin routing, verify that your origin server supports the required SSL/TLS protocols and ciphers.