Object Storage Service:Monitoring, diagnosis, and troubleshooting

Monitor the overall status

• Availability/Valid Request Proportion by User (%)

  Availability/Valid Request Proportion by User is the most important metric used to indicate OSS stability and whether OSS is correctly used. A percentage less than 100% indicates that some requests fail.

  

  A percentage less than 100% may be caused by OSS optimization such as partition migration for load balancing. In this case, OSS SDKs provide relevant retry mechanisms to handle error requests due to this temporary optimization. This way, your business is not affected.

  To determine the types and causes of error requests and troubleshoot the errors, analyze the requests based on request status details and request status distribution in the Cloud Monitor console for OSS.

  In addition, it is expected that valid request proportion may be less than 100% in some business scenarios. For example, you need to send a request to check whether an object exists before you manage the object in some scenarios. If the object does not exist, a 404 error is returned for the request, which results in a valid request proportion lower than 100%. In this case, the error is returned as expected and does not indicate an actual issue.

  For businesses that require high availability of OSS, you can configure an alert rule that is triggered when the metric falls below an expected threshold value.

• Number of Total/Valid Requests by User

  The Number of Total/Valid Requests by User metric shows the running status of OSS in the aspect of request times. If the number of valid requests is smaller than that of total requests, it indicates that some requests fail.

  

  To follow the fluctuations of the number of valid requests, especially spikes and dips, and analyze the causes, you can configure alert rules to receive timely notifications. For more information, see Alert service.

• Request Status Distribution by User

  When the availability/valid request proportion is less than 100%, or the number of valid requests is smaller than the number of total requests, you can view Request Status Distribution by User to determine the type of error requests. For more information about Request Status Distribution by User, see Metrics.

  

Monitor request status

Monitor performance

For metrics that indicate the performance of OSS, you must focus on their abnormal fluctuations such as a spike in average latency and extended high latency for requests. You can configure alert rules for performance metrics so that a notification is sent when an alert rule is triggered.

Monitor billable items

The OSS monitoring service allows you to monitor the following billable items: the storage usage, the number of the PUT requests and the GET requests, outbound traffic over the Internet, which does not include CRR outbound traffic or CDN outbound traffic. The OSS monitoring service does not support alert settings for billable items and OpenAPI reading.

The OSS monitoring service provides statistics of resources usage by user and by bucket per month. The OSS resource usage of an Alibaba Cloud account or a specific bucket is calculated and updated every hour. This way, you can calculate your storage costs during the current month.

Problem diagnosis

• Performance diagnosis

  You must determine the baseline of your application performance based on your business requirements. Note that a failed request sent from the client may be caused by one or more elements throughout the transmission link of the request, such as excessive overloads of OSS, the configurations of TCP of the client, and the traffic bottleneck from networks.

  This way, you can locate the possible issue based on the performance metric provided by the monitoring service. Then, query the logs for detailed information and diagnose the fault further.

• Error diagnosis

  The client application receives an error message from the server when a request error occurs. The monitoring service records and displays the counts and proportions of different error requests. You can check logs of the server, the client, and the network conditions to obtain the details about a specific request. Generally, the HTTP status code, the error code, and the error message in the response indicate the possible cause of an error request.

  For more information about error codes, see Error responses.

• Use the logging feature to diagnose

  OSS provides the logging feature for requests from users, which can track the details about requests throughout the transmission between the client application and the server.

  For more information about how to enable logging and use this feature, see Configure logging.

  For more information about the naming conventions and formats of OSS logs, see Logging.

• Use the logs of network conditions to diagnose

  Generally, you can determine the cause of a fault based on logs of the server and the client application. However, in some cases, to determine the cause of the fault, you also need network logs to obtain information about network conditions between the server and the client, such as the traffic and data transmitted in this process. For example, an error is reported for a user request, but no logs are generated for the request on the application server. In this case, check OSS logs to determine whether the cause is on the application user client, or use network monitoring tools to check the network conditions.

  Wireshark is one of the most common tool for log monitoring and analysis. This free tool works on the level of data packages. You can use this tool to query the details about data packages transmitted over all protocols so that you can determine whether failures are caused by package loss or network connections.

  For more information about how to use Wireshark, visit Wireshark User’s Guide.

E2E tracking and diagnosis

In a normal process, a request is sent from the client to OSS, and the OSS server processes the request and sends a response to the client. You can track the whole process between the client and the server. You can use logs of the client application, networks, and the server to locate a potential fault.

Performance-related issues

• High average E2E latency and low average server latency

  The following possible causes are inferred from the preceding description of the average E2E latency and the average server latency:

  • Slow response of client applications

    • The available connections and threads are limited.

      • For limited available connections, you can run relevant commands to check whether a large number of connections is in the TIME_WAIT state. If a large number of connections is in the TIME_WAIT state, you can adjust the number of CPU cores to handle this issue.

      • For limited threads, you can view whether bottlenecks exist in resources such as the client CPU, memory, and networks. If not, you can properly increase the number of concurrent threads.

      • If the issue persists, you must optimize the code of the application. For example, you can optimize the code of the application to support asynchronous access. You can also use the performance analysis feature to identify application features that are most commonly used and then optimize the client application.

    • Limited resources of CPU, memory, or bandwidth

      In this case, you must use the monitoring feature of relevant systems to determine resource bottlenecks. Then, optimize the code of the application to adjust the limits of the resources, or you can scale up the resources of the client such as increase the number of CPU cores or increase the memory.

  • Poor network conditions

    Generally, high average E2E latency is caused by temporary poor network conditions. You can use Wireshark to analyze accidental and persistent network issues such as the loss of data packages.

• Low average E2E latency, low average server latency, and high client request latency

  For high client request latency, the high latency most likely occurs before requests arrive at the application server. Therefore, we recommend that you analyze why requests from the client do not arrive at the server.

  The following scenarios may cause high client request latency:

  • The available connections and threads are limited.

    • For limited available connections, you can run relevant commands to check whether a large number of connections is in the TIME_WAIT state. If a large number of connections is in the TIME_WAIT state, you can adjust the number of CPU cores to handle this issue.

    • For limited threads, you can view whether bottlenecks exist in the client CPU, memory, and network resources. If not, you can properly increase the number of concurrent threads.

    • If the issue persists, you must optimize the code of the application, such as access by using the asynchronous method. You can also use the performance analysis feature to identify application features that are most commonly used and then optimize the client application.

  • Multiple retries occur for request in the client. In this case, you must analyze the cause based on the retry information. You can determine whether retries occur in the client by using the following methods:

    • Check the client logs to view whether retries have occurred. Take OSS SDK for Java as an example. You can query log prompts of the warn- or info- level: If similar logs are recorded, retries may occur in the client or in the server.

      [Server]Unable to execute HTTP request:
        or
        [Client]Unable to execute HTTP request:

    • Take OSS SDK for Java as an example. You can query the following log if the level of the client log is debug. If similar logs are recorded, retries must have occurred.

      Retrying on

  If the client has no faults, consider potential issues of networks such as the loss of data packages. You can use tools such as Wireshark to analyze the cause of network issues.

• High average server latency

  For high average server latency of downloads and uploads, consider the following possible causes:

  • A large number of clients frequently access an object.

    In this case, you can view OSS logs to determine whether an object or a group of objects is frequently accessed.

    For the scenario of downloads, we recommend that you activate Content Delivery Network (CDN) for the bucket to improve the performance and reduce the generated traffic. For the scenario of uploads, we recommend that you modify the access control list (ACL) of the bucket or the object so that users cannot write data to the bucket or the object if this does not affect your business requirements.

  • Internal issues of OSS

    For internal issues of OSS, they may not be solved by optimizing the code of the application. In this case, contact technical support to provide the client logs or the Request ID of the failed request in OSS logs.

Errors of the application server

Network errors

A network error occurs when the client cancels the request or loses the connection during a request process. If the client cancels the request, you must check the code of the application and obtain why and when the client disconnects with OSS. If the client loses the connection, you can use tools such as Wireshark to analyze possible causes.

Client errors

• Increase of client authorization error requests

  When the client authorization error request increases, or the client application receives a large amount of the HTTP status code 403, consider the following possible causes:

  • Invalid bucket domain name

    • If users use the third-level domain or the second-level domain to access the bucket, the region contained in the domain name may be different from the region in which the bucket is located. For example, the accessed bucket is located in the China (Hangzhou) region, but the accessed domain name is Bucket.oss-cn-shanghai.aliyuncs.com. In this case, you must confirm the region in which the bucket is located and correct the accessed domain name.

    • If you have enabled CDN, the origin mapped to CDN may be the wrong domain name of the bucket. In this case, check whether the origin is the third-level domain of the bucket.

    • If users use the client of JavaScript and a 403 error is returned, check whether CORS is configured for the web browser that users use to access the bucket. In this case, check the CORS settings of the bucket and correct the settings so that users can use the web browser to access the bucket. For more information about how to configure cross-origin resource sharing (CORS), see Configure CORS.

  • Access control

    • If you use the AccessKey pair of an Alibaba Cloud account to access the bucket, check the validity of your AccessKey pair.

    • If you use the AccessKey pair of a RAM user to access the bucket, check the validity or the authorization of the AccessKey pair.

    • If you use a token generated by Security Token Service (STS), check whether the temporary token expires. If the token has expired, apply for a new token.

    • If the access control list (ACL) is configured for the accessed bucket or object, check whether users are allowed to perform specific operations based on the ACL settings.

  • Expired URL

    If a 403 error occurs when the third party accesses the bucket by using a signed URL, the most possible cause is that the signed URL has expired.

  • A 403 error may occur when RAM users log on to OSS tools such as ossftp, ossbrowser, and the OSS console. In this case, check whether you enter the correct AccessKey pair or whether you have the permission to call the GetService operation if your account is a RAM user.

• Increase of 404 errors for client requests

  A 404 error for a client request indicates that the data that users access does not exist. When the number of 404 errors for client requests increases, consider the following possible causes:

  • The business logic of the application. For example, an application calls the doesObjectExist method provided by OSS SDK for Java to check whether an object exists before further actions. If the object does not exist, the value of false is returned to the client and a 404 error message is generated on the server. Therefore, in the business scenario, a 404 status code does not indicate an error.

  • The accessed object is deleted by the client application or other processes. In this case, query OSS logs for the accessed object.

  • Repeat delete operations caused by network failure. For example, the client initiates an operation to delete an object. The request arrives at the server, and the object is deleted. However, the response does not arrive at the client due to network failure. As a result, the client sends another request to delete the object, and a 404 error occurs. In this case, you can query and view the client logs and OSS logs to determine the cause of the 404 error.

    • Query the client logs and check whether a repeated request is sent from the client.

    • Query OSS logs. Then, check whether two delete operations are initiated on the object and the HTTP status code of the first operation is 2xx.

• Low valid request proportion and high client error requests

  Valid request proportion is the proportion of successful requests whose responses are the HTTP status code 2xx or 3xx to total requests. Requests whose responses are the HTTP status code 4xx or 5xx are counted as failed requests and lower the valid request proportion. Client other error requests by user refer to error requests other than server error requests, network error requests, client authorization error requests, resource not found error requests and client timeout error requests. The responses to the preceding error requests are respectively 5xx, 499, 403, 404, 408, or 400 whose corresponding OSS error code is RequestTimeout.

  You can query OSS logs to determine the error type. Then, refer to OSS error codes and solve the errors by modifying the code of the application. For more information, see Error responses.

Exceptional increase in storage usage

Other storage service issues