Real-time log query - Object Storage Service - Alibaba Cloud Documentation Center

A large number of logs are generated when Object Storage Service (OSS) resources are accessed. OSS uses Simple Log Service to help you query and collect statistics for OSS access logs and audit access to OSS, track exception events, and troubleshoot problems in the OSS console. This helps you improve efficiency and make informed decisions.

Prerequisites

Real-time log query is enabled for the bucket. For more information, see Real-time log query.
Simple Log Service is authorized to access OSS. If you have not authorized Simple Log Service to access OSS, follow the instructions on the Cloud Resource Access Authorization page to complete the authorization.

Features

Pushes logs to Simple Log Service within 3 minutes and allows you to view real-time logs in the OSS console.
Provides log analysis and common analysis reports for easy data queries.
Allows you to query and analyze raw logs in real time and filter logs by bucket, object name, API operation, or time.

Billing rules

If your Logstore uses the pay-by-feature billing mode, you are not charged log storage or index traffic fees if OSS access logs are stored for no more than 7 days and the compressed write traffic or index traffic per day is no more than 900 GB. If the size of an OSS access log is 1 KB, 900 GB is equivalent to 900 million logs. If the limits are exceeded, you are charged log storage and index traffic fees for the excess logs. The fees are included in the bills of Simple Log Service. For more information, see Billable items of pay-by-feature.
You are charged for reading data from a dedicated Logstore, processing log data, and delivering log data. These charges are included in your Simple Log Service bills.
If your Logstore uses the pay-by-feature billing mode, the real-time log query feature provides a free quota for shards per month. The free quota is calculated by using the following formula: 16 × 31. The unit of the free quota is shard-day. After the free quota is exhausted, you are charged fees, and the fees are included in the bills of Simple Log Service. For more information, see Billable items of pay-by-feature.

Procedure

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
On the Real-time Log Query tab, enter a query statement and an analysis statement based on the query syntax and analysis syntax to analyze log fields.
For example, you can analyze the operation log field to monitor the number of the GetObject operations on the bucket named examplebucket in the past seven days.
Sample query statement and analysis statement:
```
* and __topic__: oss_access_log AND bucket:examplebucket AND operation:GetObject | select
  count(*) AS GetObjectCount
```
Sample query and analysis results:
For more information, see Common examples of real-time log query.

Related operations

Enable real-time log query

You can enable real-time log query when you create a bucket or after you create a bucket.

Enable real-time log query when you create a bucket

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, click Create Bucket.
In the Create Bucket panel, turn on Real-time Log Query and configure other parameters.
For more information about how to configure other parameters, see Create buckets.
Click OK.

Enable real-time log query after you create a bucket

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
Click Enable Now.

Modify the retention period of logs

After you enable real-time log query, OSS allows you to query logs of the last seven days free of charge. You can click Set Log Retention Period in the upper-right corner to modify the retention period of logs.

Reset the index fields of logs

When you query real-time logs, fields may change. For example, a new field appears or a field is added to the index. However, these changes are not automatically reflected on the real-time log query page. You must manually refresh indexed fields by using one of the following methods.

Important

The following methods reset the index to the latest version of the built-in index and invalidate custom indexed fields. Before you reset log indexing, check whether the invalidation of custom indexed fields affects your business scenarios. If the invalidation does not affect your business scenarios, perform the following procedure to reset indexing. If the invalidation affects your business scenarios, manually add custom fields to the index. For more information, see Create indexes.

Use the OSS console

Disable real-time log query, and then enable it.

Use the Simple Log Service console

Log on to the Log Service console.
In the Log Application section, click the Cloud Service Lens tab. Then, click CloudLens for OSS.
On the Data Import Management page, click the Destination Logstore tab. Find the Logstore that stores the access logs and click Reset in the Index Reset column.
In the message that appears, click OK.
Note
Approximately 1 minute is required for index resetting to take effect.

Use LogReduce

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
Click the LogReduce tab. If you have not enabled LogReduce, click Enable LogReduce.
In the message that appears, click OK.
For more information about LogReduce, see LogReduce.

Disable real-time log query

If you no longer need to retain log data, perform the following steps to disable real-time log query:

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
Click Disable Real-time Log Query in the upper-right corner.
Important
When you enable real-time log query, Simple Log Service projects are automatically created. However, the projects are not automatically deleted when you disable real-time log query. Therefore, after real-time log query is disabled, you must log on to the Simple Log Service console to delete the projects that are automatically created to avoid unexpected charges. For more information, see Delete a project.