All Products
Search

This Product

    :How do I fix a 5xx error reported for back-to-origin routing?

    Document Center

    :How do I fix a 5xx error reported for back-to-origin routing?

    Last Updated:Mar 08, 2024

    Problem description

    When you use an accelerated domain name to access resources, the request is sent to Alibaba Cloud CDN points of presence (POPs), and if a cache miss occurs, the request is redirected to the origin server. If an error is reported in the process, network routing is a likely cause. If a back-to-origin routing exception occurs, a 5xx error is returned. Common 5xx errors include 502 Bad Gateway, 503 Service Temporarily Unavailable, and 504 Gateway Time-out.

    Causes

    Check the X-Swift-Error field in the response headers for more information about the 5xx error. The X-Swift-Error field contains error information such as forward retry timeout and orig response 5xx error. Meanwhile, use cURL or the Network tab in browser developer mode to check the response time.

    • If a 5xx error occurs globally, a common cause is incorrect Alibaba Cloud CDN configurations or origin server exceptions:

      • The origin server is inaccessible or the domain name of the origin server cannot be resolved.

      • Back-to-origin routing over HTTPS is configured in Alibaba Cloud CDN, but the origin server does not support HTTPS.

      • Server Name Indication (SNI) is enabled for the origin server but not enabled in Alibaba Cloud CDN.

    • If this error occurs occasionally or only in specific regions, it is likely caused by the network condition on the origin server or other factors, such as the following ones:

      • The security policies configured on the origin server block IP addresses of the POPs.

      • The network connection to the origin server is unstable, the cross-border back-to-origin route is unstable, or the response speed of the dynamic interface is unstable.

    Solutions

    Alibaba Cloud CDN is a global network of POPs that are distributed across the globe. Alibaba Cloud CDN caches resources on origin servers to POPs to accelerate access. When a request is sent to access a resource for the first time or access an expired resource cached on POPs, the request is redirected to the origin server. The response time of the origin server and the time to first byte (TTFB) are crucial to the back-to-origin routing process. If a 5xx error is reported for back-to-origin routing, use the following methods to troubleshoot the error:

    Scenario 1: The origin server is a local host or an ECS instance

    If your origin server is a local host or an Elastic Compute Service (ECS) instance, perform the following steps:

    Step 1: Obtain HTTP response headers

    Use cURL or Wget to obtain HTTP response headers. The following example shows how to use the curl command to obtain HTTP response headers.

    Note

    You can also press F12 to open the developer tool of the browser and go to the Network tab to locate the URL for which the 5xx error is returned. 

    curl -vo [$File_Name] [$IP]
    Note

    • [$File_Name]: the name of the file created in the current directory. The output of the website that sent the request is recorded in the file.

    • [$IP]: the IP address of the website that sent the request.

    If the X-Swift-Error response field has a value of "orig response 5xx error" or a similar value, the back-to-origin routing timed out or request processing on the origin server timed out.

    image.png

    Check the following aspects based on the retrieved HTTP response.

    Step 2: Test and analyze network performance based on the regions affected

    Use Networkbench or 17CE to check the network performance of POPs. Use different troubleshooting methods based on the regions where the error occurs:

    • Single region: The cause of the error may be related to the connection between the POP and origin server. Perform a network test several times to check whether the network connection is restored. If this error causes significant consequences, record the response headers and contact Alibaba Cloud technical support.

    • Several regions: Run the following command to test the quality of connections between the client and origin server and between the client and POPs. 

      time telnet [$Test_IP] [$Test_Port]
ping [$Test_IP]
      Note

      • The first command queries the time consumed to execute the telnet command.

      • The second command tests network connectivity.

      • [$Test_IP]: the IP address. It can be the IP address of the origin server or the IP address of a POP.

      • [$Test_Port]: the port number. It can be the port number of the origin server or the port number of a POP.

    Step 3: Check the bandwidth and QPS of the domain name

    Use the following steps to check the bandwidth and QPS of the domain name.

    1. Log on to the Alibaba Cloud CDN console.

    2. In the left-side navigation pane, choose Content Delivery > Monitoring & Usage Analytics > Real-time Monitoring to check the recent back-to-origin bandwidth and QPS. A bandwidth or QPS spike may cause response timeouts of the origin server.

    3. In case of a back-to-origin bandwidth spike, check whether new resources are added to the origin server or whether cache rules are configured for frequently accessed resources. If your business involves a large quantity of gaming data or images, we recommend that you deploy resources to Alibaba Cloud CDN in advance by using the prefetch feature to prevent performance bottlenecks on the origin server caused by back-to-origin routing bandwidth spikes. We also recommend that you configure a long cache time-to-live (TTL) for frequently accessed resources.

    4. Check whether cache rules are configured on Alibaba Cloud CDN and the origin server. If the no-cache response header is configured for the origin server, it takes precedence over the cache configuration on Alibaba Cloud CDN. If no cache rules are configured for the origin server and Alibaba Cloud CDN POPs, the default TTL is 10 to 3,600 seconds. For more information, see What is caching?

      If you are a new user of Alibaba Cloud CDN or not familiar with Alibaba Cloud CDN caching, we recommend that you configure a maximum TTL for directories and configure a separate policy for resources that do not require caching. You can then specify priorities to control matching. For more information, see Create a cache rule for resources.

    Step 4: Check the connectivity and domain name resolution of the origin server

    Alibaba Cloud CDN POPs are on the Internet. The origin server must be connected to the Internet for successful back-to-origin routing. If the configured IP address for the origin server is inaccessible over the Internet, the port on the origin server is inaccessible, or the domain name of the origin server is not resolved, a 5XX error is reported on the origin server.

    1. The origin server is inaccessible. You can run the ping command to test the connectivity of the origin server. 

      ping [$IP]
      Note

      [$IP] is the IP address of the origin server.

    2. The port configured on the origin server is inaccessible, or the origin server directly returns a 5xx error. A port test using telnet returns the Connection timed out error.

      1. If port 80 is configured on the origin server, run the following command to test the port status. 

        telnet [$IP] 80

      2. If port 443 or a custom port is configured on the origin server, test whether port 443 or the custom port is accessible.

      3. Obtain the IP address and port configurations of the origin server in the Alibaba Cloud CDN console and use the local hosts file to map the IP address to the origin server. Perform Layer 7 testing to check whether the origin server fails to respond or directly returns a 5xx error. Alternatively, run curl to test the port status on the origin server.

        • If the port on the origin server is port 80, run curl -voa http://[$Domain] -x [$IP]:80.

          Note

          [$Domain] is the accelerated domain name.

        • If the port on the origin server is port 443, run curl -voa https://[$Domain] --resolve [$Domain]:443:[$IP].

    3. If the domain name of the origin server is configured in Alibaba Cloud CDN, but domain name resolution is not configured, back-to-origin routing fails. You can use ping or nslookup to test whether the domain name of the origin server is normally resolved. For example, if pinging the domain name example.aliyundoc.com of the origin server returns the unknown host error or running the nslookup command to query the domain name resolution record of the origin server returns server can't find example.aliyundoc.com: NXDOMAIN, the domain name of the origin server is not resolved. For more information about how to configure domain name resolution, see Add a CNAME record for a domain name.

    Step 5: Check back-to-origin configurations

    If back-to-origin routing over HTTPS is configured in Alibaba Cloud CDN, but the origin server does not support HTTPS, a 5xx error is returned for back-to-origin routing. Use the following methods to troubleshoot the error based on your business scenario.

    1. Port 443 is configured on the origin server, but the origin server does not support HTTPS. If the port for the origin server is port 443 on the back-to-origin page of the Alibaba Cloud CDN console, back-to-origin routing uses HTTPS over port 443. Port 443 must be opened on the origin server, and the origin server must have an HTTPS certificate.

      • If the origin server does not support HTTPS, the origin server returns a 5xx error, which indicates that back-to-origin routing failed. In this case, you need to change the back-to-origin port to port 80.

      • If your business requires back-to-origin routing over port 443, configure an HTTPS certificate for the origin server.

    2. Back-to-origin routing is configured to follow the protocol used by the client, but the origin server does not support HTTPS. If back-to-origin routing is configured to use HTTPS, Alibaba Cloud CDN uses HTTPS for back-to-origin routing. If the protocol for back-to-origin routing is set to Follow, and the client uses HTTPS to request resources, Alibaba Cloud CDN uses HTTPS for back-to-origin routing. If the origin server does not support HTTPS, an access failure occurs. In this case, disable the protocol-following back-to-origin routing or use HTTP as the protocol for back-to-origin routing.

    3. Use the curl command to test the accessibility of the origin server. 

      curl -voa https://[$Domain] --resolve [$Domain]:443:[$IP]

    4. You can also modify the local hosts file to map it to the origin server and use a browser to initiate HTTPS requests to test the accessibility of the origin server. If the Your connection is not private error is returned, access over HTTPS is not supported by the origin server.

    Scenario 2: Your business application uses the combination of Alibaba Cloud CDN and OSS

    If your business application combines Alibaba Cloud CDN and Object Storage Service (OSS), perform the following steps to troubleshoot the error:

    1. Use the --probe-item option of the ossutil tool to check whether the outbound bandwidth exceeds 10 Gbit/s and whether the QPS exceeds 10,000. If the limits are exceeded, OSS traffic throttling is triggered, and processing of back-to-origin requests beyond the limits is delayed. For more information about traffic throttling, see Limits.

      Note

      • For more information about how to use the --probe-item option in ossutil, see View options.

      • Back-to-origin requests sent in case of OSS traffic throttling do not return an unavailability error. They are processed in a delayed manner.

    2. Check whether the outbound traffic on Alibaba Cloud CDN exceeds the bandwidth limit of the origin server. If the origin server reaches the maximum bandwidth, network and application layer issues occur.

      Note

      • You can use the iperf tool to check the traffic between two POPs.

      • You can use the Netstat tool to check the number of network connections.

    3. Check whether the origin server has firewall restrictions. Specifically, check whether the origin server limits the speed or IP addresses for back-to-origin requests.

    4. If the previous troubleshooting steps do not indicate any exceptions, check the Web error logs of the origin server for 5xx error messages. If no errors are recorded in the log files at the application layer, test the connection from the origin server to POPs to check for 5xx errors caused by network jitters. This check process is subject to the actual error frequency and business status.

      Note

      Generally, if the origin server easily reaches the maximum bandwidth or packet loss or packet latency occurs regularly, we recommend that you increase the value of the send timeout or read timeout parameter in the nginx.conf file.