Server Load Balancer:Common ALB status codes and possible causes

Possible causes:

• A backend server returns the HTTP 400 status code, and ALB passes the backend status code to the client. Check why the backend server returns the HTTP 400 status code.

• The format of the HTTP header is invalid. For example, the length of the content is inconsistent with the length of the request body. We recommend that you analyze the format of the HTTP request on the client and compare it with a valid request.

• An HTTPS listener is added to the ALB instance. If the client does not send an HTTPS request but the HTTP 400 status code is returned, we recommend that you check whether the client sent an HTTP request to an HTTPS port.

• Before the HTTP request is sent, the client closes the connection. We recommend that you capture packets and identify the cause of the disconnection.

• ALB requires that each HTTP header does not exceed 32 KB in length. If this limit is exceeded, the HTTP 400 status code is returned. We recommend that you adjust the length of the HTTP header.

Possible causes:

• The backend server does not support the request method used by the client. We recommend that you check whether the method used by the client matches the method used by the backend server. For example, to check whether a specific method is supported by a backend server, run the curl -X method ip:port command. method specifies the request method that is used by the client, ip specifies the IP address of the backend server, and port specifies the port of the backend server.

• ALB does not support the TRACE request method. We recommend that you use another method.

Possible causes:

• During the request timeout period, only part of the request is sent, and ALB closes the connection. For example, only the HTTP header is sent and the body is not sent. We recommend that you capture packets to check whether the client has performance bottlenecks or other abnormal behaviors. You can also increase the request timeout period of the ALB listener to check whether the issue can be resolved. For more information, see Modify a listener.

• Errors exist on the connection between the client and the ALB instance. The round-trip time (RTT) of TCP packets is excessively large or packets are lost. We recommend that you check the request_time and tcpinfo_rtt fields of the access log or capture packets to check whether the client network contains errors.

• The amount of network traffic to the ALB instance is excessively large, which causes bandwidth throttling and packet loss. We recommend that you check the outbound bandwidth and the number of dropped connections of the ALB instance by using CloudMonitor. For more information, see ALB monitoring metrics and View the monitoring information about an ALB instance.

Possible causes:

• A backend server returns the HTTP 414 status code, and ALB passes the backend status code to the client. Check why the backend server returns the HTTP 414 status code.

• The URI length of requests to ALB cannot exceed 32 KB. If the URI of a request exceeds 32 KB, the HTTP 414 status code is returned. We recommend that you shorten the URI. If the amount of data to be transmitted is large, we recommend that you use the POST method to transmit data. The body of a POST request to ALB can be up to 50 GB in size.

Possible causes:

• Errors exist on the connection between the client and the ALB instance. The RTT is large or packets are lost. We recommend that you check the request_time and tcpinfo_rtt fields of the access log or capture packets to check whether the client network contains errors.

• The amount of network traffic to the ALB instance is excessively large, which causes bandwidth throttling and packet loss. We recommend that you check the outbound bandwidth and the number of dropped connections of the ALB instance by using CloudMonitor. For more information, see ALB monitoring metrics and View the monitoring information about an ALB instance.

• The request processing time of the backend server is too long, and exceeds the request timeout period of the client. The upstream_response_time field in the access log indicates the time that is required for the backend server to process the request. We recommend that you check the CPU, memory, and network of the backend server for performance bottlenecks.

• The request timeout period set on the client is too short. As a result, the client closes the connection before the HTTP request is sent. We recommend that you check the request_time field in the access log. This field indicates the time that is required for the client to send the request. Refer to the value of this field to set a proper request timeout period on the client.

• The client encounters an unknown issue and closes the connection before the HTTP request is sent. We recommend that you check whether the client closes the connection before the HTTP request is sent.

Possible causes:

• A backend server returns the HTTP 500 status code, and ALB passes the backend status code to the client. Check why the backend server returns the HTTP 500 status code.

• The backend server closes the connection before sending the response. We recommend that you capture packets on the backend server to determine the cause and troubleshoot.

Possible causes:

• The backend server returns the HTTP 502 status code, and ALB passes the backend status code to the client. Check why the backend server returns the HTTP 502 status code.

• The ALB backend server returns another HTTP status code (such as 504 and 444), but ALB returns the HTTP 502 status code. We recommend that you check the upstream_status and status fields in the access log or capture packets to check whether the backend server has errors.

• The communication between ALB and the backend server over TCP contains errors. Check whether the backend server is healthy, whether the service port is listened by listeners, or whether TCP handshakes succeed.

• The backend server backlog is full, which causes packets to be dropped. We recommend that you use netstat to check whether the network statistics of the backend server show the number of dropped packets. For example, you can run the netstat -s | grep -i listen command.

• The length of the packets sent by the client exceeds the maximum transmission unit (MTU) of the backend server. In this case, the health check is successful, or shorter packets are sent as expected. However, longer packets are not sent as expected. We recommend that you capture packets on the backend server to analyze whether the packet length meets the requirements.

• The packets returned by the backend server are in an invalid format or contain invalid HTTP headers. We recommend that you capture packets on the backend server to check whether the format of HTTP responses is invalid.

• A backend server of ALB has not completed the processing of the request. Check the log data of the backend server. In addition, check the CPU and memory usage.

Possible causes:

• A backend server returns the HTTP 503 status code, and ALB passes the backend status code to the client. Check why the backend server returns the HTTP 503 status code.

• The network traffic from clients triggers ALB throttling. You can use one of the following methods to troubleshoot the issue:

  • View the number of requests per second in the CloudMonitor console. For more information, see ALB monitoring metrics and View the monitoring information about an ALB instance.

  • CloudMonitor displays monitoring data at the minute level. You can view the number of requests per second by using the access log. View the upstream_status field of the access log. If the content is "-", the request is not sent to the backend server.

  • Check whether the response header contains the ALB-QPS-Limited:Limited field. If so, it indicates that the request has triggered ALB throttling.

• The client accesses the IP address of the ALB instance instead of using the domain name when the client accesses ALB, or does not update the DNS resolution results when the client uses the domain name to access ALB. As a result, requests cannot be distributed among multiple ALB IP addresses and the HTTP 503 error code is returned. We recommend that clients access the domain name of ALB instead of the IP address. In addition, make sure that clients use different IP addresses returned by DNS to access ALB.

• No backend server is associated with the ALB listener or the weight of the backend server is set to 0.

Possible causes:

• If the 504 status code is returned by the backend server, check whether the backend server is overloaded.

• The connection from ALB to the backend server timed out. The timeout period is set to 5 seconds by default. You can check whether the upstream_connect_time field in the access log is set to 5 seconds or approximately 5 seconds. We recommend that you capture packets to determine the cause of the response timeout.

• The workload on the backend server is heavy and the time that is required to return a response is longer than the specified timeout period. For example, the request timeout period is set to 60 seconds. If the response time is 60.001 seconds, ALB returns the HTTP 504 status code. You can view the network latency during the period in which issues occur in the CloudMonitor console or in the access log. To view the network latency, check the upstream_rt field in CloudMonitor, or the upstream_response_time field in the access log.