Common examples of real-time OSS log query - Object Storage Service

By using the real-time log query feature provided by Simple Log Service, you can analyze the access logs, batch deletion logs, and hourly measurement logs of Object Storage Service (OSS). This way, you can monitor and optimize the usage of OSS in a timely manner, locate and analyze potential problems, and improve the efficiency of data storage and access. This topic describes common examples of real-time log query.

Analyze the outbound Internet traffic of objects in an OSS directory

Important

The host field in a request can be forged. The following query results are only for estimation and cannot be used as a basis for billing. The actual bills for OSS outbound Internet traffic take precedence.

By using the OSS usage query, you can view the outbound Internet traffic of objects in an OSS bucket, but you cannot view the outbound Internet traffic of objects in an OSS directory. If you want to analyze the outbound Internet traffic of objects in an OSS directory, you can use the host field in OSS access logs to query the OSS Internet endpoint. Then, you can use the sync_request field to exclude Alibaba Cloud Content Delivery Network (CDN) back-to-origin requests, match the prefix of the object field with objects in the OSS directory, and then calculate the sum of the content_length_out fields. This way, you can obtain the total amount of data transferred from objects in the directory to the Internet.

Query statement

The following code provides a query statement that is used to calculate the outbound Internet traffic of objects in the exampledir directory of the examplebucket bucket.

* and __topic__ : oss_access_log and bucket: examplebucket and host : "examplebucket.oss-cn-hangzhou.aliyuncs.com" not sync_request : cdn | select
  SUM(content_length_out) AS total_traffic_out_byte
WHERE
  url_decode(object) LIKE 'exampledir/%'

Query result

The query result indicates that the total outbound Internet traffic of objects in the exampledir directory of the examplebucket bucket is 11,749 bytes.

Analyze the changed size of objects in an OSS directory

By using the OSS usage query, you can view the total changed size of objects in an OSS bucket, but you cannot view the total changed size of objects in an OSS directory. If you want to analyze the total changed size of objects in an OSS directory, you can match the prefix of the object field in OSS access logs to collect statistics on the objects in the OSS directory. Then, you can calculate the sum of the delta_data_size fields. This way, you can obtain the total changed size of objects in the directory.

Query statement

The following code provides a query statement that is used to calculate the total changed size of objects in the exampledir directory of the examplebucket bucket.

* and __topic__ : oss_access_log and bucket: examplebucket | select
  SUM(delta_data_size) AS total_delta_data_size
WHERE
  url_decode(object) LIKE 'exampledir/%'

Query result

The query result indicates that the total size of objects in the exampledir directory of the examplebucket bucket has increased by 941 bytes.

Analyze the OSS Internet requests that are not accelerated by CDN

Important

After you enable CDN acceleration, if you still view significant OSS Internet requests, the URLs of resources referenced in your business still point to the OSS origin server instead of the CDN acceleration endpoint. To identify Internet requests that are not accelerated by CDN, you can use the host field in OSS access logs to filter the OSS Internet endpoint and use the sync_request field to exclude CDN back-to-origin requests. You can also determine the web pages or applications from which these requests are sent based on the referer field.

Query statement

The following code provides a query statement that is used to analyze OSS Internet requests that are not accelerated by CDN in the examplebucket bucket.

* and __topic__: oss_access_log and bucket: examplebucket and host : "examplebucket.oss-cn-hangzhou.aliyuncs.com" not sync_request: cdn | select
  referer,
  host,
  count(*) as request_count
group by
  referer,
  host
order by request_count desc

Query result

The query result indicates that the top three sources of the OSS Internet requests that are not accelerated by CDN in the examplebucket bucket are: requests in which the Referer field is left empty, and requests from websites whose domain names end with .com or .vip. If you enter the URL in the address bar of the browser, you may leave the Referer field empty in the request.