A large number of logs are generated when Object Storage Service (OSS) resources are accessed. After you enable and configure logging for a bucket, OSS generates access logs every hour based on predefined naming rules and then stores the logs in a specific bucket. You can use Simple Log Service or build a Spark cluster to analyze logs.
Usage notes
The source bucket for which logs are generated and the destination bucket in which the logs are stored can be the same bucket or different buckets. However, the destination bucket must belong to the same Alibaba Cloud account and be located in the same region as the source bucket.
Logs are generated within 48 hours. Requests received within a specific one-hour period may be recorded in the log objects for the previous or next hour. As a result, the log objects for a specific period of time cannot ensure the integrity and immediacy of logging for that period of time.
Before you disable logging, OSS generates log objects every hour. We recommend that you delete log objects that you no longer need to reduce storage costs.
To maintain OSS-HDFS availability and prevent data contamination, do not set Log Prefix to .dlsdata/ when you configure logging for a bucket for which OSS-HDFS is enabled.
OSS adds specific fields to the end of logs. We recommend that you consider potential compatibility issues when you develop log processing tools.
The point in time when OSS received the access request.
RequestURL
GET /example.jpg HTTP/1.0
The request URL that contains a query string.
OSS ignores a query string parameter that starts with x-. However, this parameter is recorded in logs. You can tag a request by using a query string parameter that starts with x- and use the tag to identify the log of the request.
HTTPStatus
200
The HTTP status code that is returned by OSS.
SentBytes
999131
The outbound traffic generated by the request. Unit: bytes.
RequestTime
127
The amount of time consumed by the request. Unit: milliseconds.
Referer
http://www.aliyun.com/product/oss
The Referer header in the HTTP request.
UserAgent
curl/7.15.5
The User-Agent header in the HTTP request.
HostName
examplebucket.oss-cn-hangzhou.aliyuncs.com
The domain name to access.
RequestID
5FF16B65F05BC932307A3C3C
The request ID.
LoggingFlag
true
Indicates whether logging is enabled. Values:
true
false
RequesterAliyunID
16571836914537****
The user ID of the requester. The value - indicates anonymous access.
Operation
GetObject
The type of the request.
BucketName
examplebucket
The name of the bucket to access.
ObjectName
example.jpg
The name of the object to access.
ObjectSize
999131
The size of the object to access. Unit: bytes.
ServerCostTime
88
The amount of time that is consumed by OSS to process the request. Unit: milliseconds.
ErrorCode
-
The error code that OSS returns. The value - indicates that no error code is returned.
RequestLength
302
The length of the request. Unit: bytes.
UserID
16571836914537****
The ID of the bucket owner.
DeltaDataSize
-
The size change of the object. The value - indicates that this request does not involve write operations on the object.
The storage class of the object to access. Values:
Standard: The storage class is Standard.
IA: The storage class is Infrequent Access (IA).
Archive: The storage class is Archive.
Cold Archive: The storage class is Cold Archive.
DeepCold Archive: The storage class is Deep Cold Archive.
-: The storage class of the object is not obtained.
TargetStorageClass
-
The storage class to which the object is moved based on a lifecycle rule or by calling the CopyObject operation. Values:
Standard: The object is moved to Standard.
IA: The object is moved to IA.
Archive: The object is moved to Archive.
Cold Archive: The object is moved to Cold Archive.
DeepCold Archive: The object is moved to Deep Cold Archive.
-: No storage class conversion is involved.
TransmissionAccelerationAccessPoint
-
The region from which the requester accessed the destination bucket by using an acceleration endpoint. For example, if the requester accesses the bucket by using the acceleration endpoint from the China (Hangzhou) region, the value of TransmissionAccelerationAccessPoint is cn-hangzhou.
The value - indicates that no acceleration endpoint is used or the requester accesses the destination bucket from the same region as the destination bucket
AccessKeyID
LTAI4FrfJPUSoKm4JHb5****
The AccessKey ID of the requester.
If the request is initiated from the OSS console, a temporary AccessKey ID that starts with TMP is displayed.
If the request is initiated based on a long-term AccessKey pair by using a tool or OSS SDK, the AccessKey ID of the AccessKey pair is displayed. Example value: LTAI4FrfJPUSoKm4JHb5****.
If the request is initiated based on temporary access credentials provided by Security Token Service (STS), the temporary AccessKey ID that starts with STS is displayed.
Note
The value - indicates that the request is anonymous.
After log objects are stored in the specified bucket in OSS, you can use Simple Log Service to analyze the log objects. Before you analyze log objects, you must import the log objects to Simple Log Service. For more information about how to import data, see Import data from OSS to Simple Log Service. For more information about the analysis feature of Simple Log Service, see Log analysis overview.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Logging.
On the Logging tab, turn on Logging and specify a bucket to store logs and a log name prefix.
Log Storage Bucket: Select a bucket from the drop-down list. You can select the same bucket for which logging is enabled or a different bucket that is located in the same region as the bucket for which logging is enabled within the same Alibaba Cloud account.
Log Prefix: Enter the directory that stores logs. If you specify this parameter, the logs are stored in the specified directory of the specified bucket. If you do not specify this parameter, the logs are stored in the root directory of the specified bucket. For example, if you enter log/ as the value of the Log Prefix parameter, the log objects are stored in the log/ directory.
Click Save.
Use OSS SDKs
The following sample code provides examples on how to configure logging by using OSS SDKs for common programming languages. For more information about how to configure logging by using OSS SDKs for other programming languages, see Overview.
import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.SetBucketLoggingRequest;
public class Demo {
public static void main(String[] args) throws Exception {
// In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint.
String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
// Specify the name of the source bucket for which you want to enable logging. Example: examplebucket.
String bucketName = "examplebucket";
// Specify the name of the destination bucket in which you want to store the log objects. The source bucket and the destination bucket can be the same bucket or different buckets.
String targetBucketName = "yourTargetBucketName";
// Set the directory in which you want to store the log objects to log/. If you specify this parameter, the log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, the log objects are stored in the root directory of the destination bucket.
String targetPrefix = "log/";
// Create an OSSClient instance.
OSS ossClient = new OSSClientBuilder().build(endpoint, credentialsProvider);
try {
SetBucketLoggingRequest request = new SetBucketLoggingRequest(bucketName);
request.setTargetBucket(targetBucketName);
request.setTargetPrefix(targetPrefix);
ossClient.setBucketLogging(request);
} catch (OSSException oe) {
System.out.println("Caught an OSSException, which means your request made it to OSS, "
+ "but was rejected with an error response for some reason.");
System.out.println("Error Message:" + oe.getErrorMessage());
System.out.println("Error Code:" + oe.getErrorCode());
System.out.println("Request ID:" + oe.getRequestId());
System.out.println("Host ID:" + oe.getHostId());
} catch (ClientException ce) {
System.out.println("Caught an ClientException, which means the client encountered "
+ "a serious internal problem while trying to communicate with OSS, "
+ "such as not being able to access the network.");
System.out.println("Error Message:" + ce.getMessage());
} finally {
if (ossClient != null) {
ossClient.shutdown();
}
}
}
}
<?php
if (is_file(__DIR__ . '/../autoload.php')) {
require_once __DIR__ . '/../autoload.php';
}
if (is_file(__DIR__ . '/../vendor/autoload.php')) {
require_once __DIR__ . '/../vendor/autoload.php';
}
use OSS\Credentials\EnvironmentVariableCredentialsProvider;
use OSS\OssClient;
use OSS\CoreOssException;
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
$provider = new EnvironmentVariableCredentialsProvider();
// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
$endpoint = "yourEndpoint";
// Specify the name of the source bucket for which you want to enable logging. Example: examplebucket.
$bucket= "examplebucket";
$option = array();
// Specify the name of the destination bucket in which the log objects are stored.
$targetBucket = "destbucket";
// Specify the directory in which the log objects are stored. If you specify this parameter, the log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, the log objects are stored in the root directory of the destination bucket.
$targetPrefix = "log/";
try {
$config = array(
"provider" => $provider,
"endpoint" => $endpoint,
);
$ossClient = new OssClient($config);
// Enable logging for the source bucket.
$ossClient->putBucketLogging($bucket, $targetBucket, $targetPrefix, $option);
} catch (OssException $e) {
printf(__FUNCTION__ . ": FAILED\n");
printf($e->getMessage() . "\n");
return;
}
print(__FUNCTION__ . ": OK" . "\n");
const OSS = require('ali-oss')
const client = new OSS({
// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to oss-cn-hangzhou.
region: 'yourregion',
// Obtain access credentials from environment variables. Before you run the sample code, make sure that you have configured environment variables OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET.
accessKeyId: process.env.OSS_ACCESS_KEY_ID,
accessKeySecret: process.env.OSS_ACCESS_KEY_SECRET,
// Specify the name of the bucket.
bucket: 'yourbucketname'
});
async function putBucketLogging () {
try {
const result = await client.putBucketLogging('bucket-name', 'logs/');
console.log(result)
} catch (e) {
console.log(e)
}
}
putBucketLogging();
# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import BucketLogging
# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com.
# Specify the name of the bucket. Example: examplebucket.
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'examplebucket')
# Specify that the generated log objects are stored in the current bucket.
# Set the directory in which the log objects are stored to log/. If you specify this parameter, the log objects are stored in the specified directory of the bucket. If you do not specify this parameter, the log objects are stored in the root directory of the bucket.
# Enable logging for the bucket.
logging = bucket.put_bucket_logging(BucketLogging(bucket.bucket_name, 'log/'))
if logging.status == 200:
print("Enable access logging")
else:
print("request_id:", logging.request_id)
print("resp : ", logging.resp.response)
using Aliyun.OSS;
using Aliyun.OSS.Common;
// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint.
var endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
// Specify the name of the source bucket for which you want to enable logging. Example: examplebucket.
var bucketName = "examplebucket";
// Specify the name of the destination bucket in which the log objects are stored. The source bucket and the destination bucket can be the same bucket or different buckets.
var targetBucketName = "destbucket";
// Create an OSSClient instance.
var client = new OssClient(endpoint, accessKeyId, accessKeySecret);
try
{
// Specify that the log objects are stored in the log/ directory. If you specify this parameter, the log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, log objects are stored in the root directory of the destination bucket.
var request = new SetBucketLoggingRequest(bucketName, targetBucketName, "log/");
// Enable logging for the source bucket.
client.SetBucketLogging(request);
Console.WriteLine("Set bucket:{0} Logging succeeded ", bucketName);
}
catch (OssException ex)
{
Console.WriteLine("Failed with error info: {0}; Error info: {1}. \nRequestID:{2}\tHostID:{3}",
ex.ErrorCode, ex.Message, ex.RequestId, ex.HostId);
}
catch (Exception ex)
{
Console.WriteLine("Failed with error info: {0}", ex.Message);
}
PutBucketLoggingRequest request = new PutBucketLoggingRequest();
// Specify the name of the source bucket for which you want to enable logging.
request.setBucketName("yourSourceBucketName");
// Specify the name of the destination bucket in which you want to store the logs.
// The source bucket and the destination bucket must be located in the same region. The source bucket and the destination bucket can be the same bucket or different buckets.
request.setTargetBucketName("yourTargetBucketName");
// Specify the directory in which the logs are stored.
request.setTargetPrefix("<yourTargetPrefix>");
OSSAsyncTask task = oss.asyncPutBucketLogging(request, new OSSCompletedCallback<PutBucketLoggingRequest, PutBucketLoggingResult>() {
@Override
public void onSuccess(PutBucketLoggingRequest request, PutBucketLoggingResult result) {
OSSLog.logInfo("code::"+result.getStatusCode());
}
@Override
public void onFailure(PutBucketLoggingRequest request, ClientException clientException, ServiceException serviceException) {
OSSLog.logError("error: "+serviceException.getRawMessage());
}
});
task.waitUntilFinished();
package main
import (
"fmt"
"os"
"github.com/aliyun/aliyun-oss-go-sdk/oss"
)
func main() {
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
provider, err := oss.NewEnvironmentVariableCredentialsProvider()
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Create an OSSClient instance.
// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint.
client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
// Specify the name of the source bucket for which you want to enable logging. Example: examplebucket.
bucketName := "examplebucket"
// Specify the name of the destination bucket in which you want to store the log objects. The source and destination buckets can be the same bucket or different buckets, but they must be located in the same region.
targetBucketName := "destbucket"
// Set the directory in which you want to store the log objects to log/. If you specify this parameter, the log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, the log objects are stored in the root directory of the destination bucket.
targetPrefix := "log/"
// Enable logging for the bucket.
err = client.SetBucketLogging(bucketName, targetBucketName, targetPrefix, true)
if err != nil {
fmt.Println("Error:", err)
os.Exit(-1)
}
}
#include <alibabacloud/oss/OssClient.h>
using namespace AlibabaCloud::OSS;
int main(void)
{
/* Initialize information about the account that is used to access OSS. */
/* Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. */
std::string Endpoint = "yourEndpoint";
/* Specify the name of the source bucket for which you want to enable logging. Example: examplebucket. */
std::string BucketName = "examplebucket";
/* Specify the name of the destination bucket in which the log objects are stored. The source bucket and the destination bucket can be the same bucket or different buckets. */
std::string TargetBucketName = "destbucket";
/* Set the directory in which the log objects are stored to log/. If you specify this parameter, the log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, the log objects are stored in the root directory of the destination bucket. */
std::string TargetPrefix ="log/";
/* Initialize resources such as network resources. */
InitializeSdk();
ClientConfiguration conf;
/* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
auto credentialsProvider = std::make_shared<EnvironmentVariableCredentialsProvider>();
OssClient client(Endpoint, credentialsProvider, conf);
/* Enable logging for the bucket. */
SetBucketLoggingRequest request(BucketName, TargetBucketName, TargetPrefix);
auto outcome = client.SetBucketLogging(request);
if (!outcome.isSuccess()) {
/* Handle exceptions. */
std::cout << "SetBucketLogging fail" <<
",code:" << outcome.error().Code() <<
",message:" << outcome.error().Message() <<
",requestId:" << outcome.error().RequestId() << std::endl;
return -1;
}
/* Release resources such as network resources. */
ShutdownSdk();
return 0;
}
#include "oss_api.h"
#include "aos_http_io.h"
/* Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. */
const char *endpoint = "yourEndpoint";
/* Specify the name of the bucket. Example: examplebucket. */
const char *bucket_name = "examplebucket";
/* Specify the name of the destination bucket in which the log objects are stored. The source bucket and the destination bucket can be the same bucket or different buckets. */
const char *target_bucket_name = "yourTargetBucketName";
/* Specify the directory in which the log objects are stored. If you specify this parameter, the log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, the log objects are stored in the root directory of the destination bucket. */
const char *target_logging_prefix = "yourTargetPrefix";
void init_options(oss_request_options_t *options)
{
options->config = oss_config_create(options->pool);
/* Use a char* string to initialize data of the aos_string_t type. */
aos_str_set(&options->config->endpoint, endpoint);
/* Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. */
aos_str_set(&options->config->access_key_id, getenv("OSS_ACCESS_KEY_ID"));
aos_str_set(&options->config->access_key_secret, getenv("OSS_ACCESS_KEY_SECRET"));
/* Specify whether to use CNAME. The value 0 indicates that CNAME is not used. */
options->config->is_cname = 0;
/* Specify network parameters, such as the timeout period. */
options->ctl = aos_http_controller_create(options->pool, 0);
}
int main(int argc, char *argv[])
{
/* Call the aos_http_io_initialize method in main() to initialize global resources, such as network resources and memory resources. */
if (aos_http_io_initialize(NULL, 0) != AOSE_OK) {
exit(1);
}
/* Create a memory pool to manage memory. aos_pool_t is equivalent to apr_pool_t. The code used to create a memory pool is included in the APR library. */
aos_pool_t *pool;
/* Create a memory pool. The value of the second parameter is NULL. This value indicates that the pool does not inherit other memory pools. */
aos_pool_create(&pool, NULL);
/* Create and initialize options. This parameter includes global configuration information, such as endpoint, access_key_id, access_key_secret, is_cname, and curl. */
oss_request_options_t *oss_client_options;
/* Allocate the memory resources in the memory pool to the options. */
oss_client_options = oss_request_options_create(pool);
/* Initialize oss_client_options. */
init_options(oss_client_options);
/* Initialize the parameters. */
aos_string_t bucket;
oss_logging_config_content_t *content;
aos_table_t *resp_headers = NULL;
aos_status_t *resp_status = NULL;
aos_str_set(&bucket, bucket_name);
content = oss_create_logging_rule_content(pool);
aos_str_set(&content->target_bucket, target_bucket_name);
aos_str_set(&content->prefix, target_logging_prefix);
/* Enable logging for the source bucket. */
resp_status = oss_put_bucket_logging(oss_client_options, &bucket, content, &resp_headers);
if (aos_status_is_ok(resp_status)) {
printf("put bucket logging succeeded\n");
} else {
printf("put bucket logging failed, code:%d, error_code:%s, error_msg:%s, request_id:%s\n",
resp_status->code, resp_status->error_code, resp_status->error_msg, resp_status->req_id);
}
/* Release the memory pool. This operation releases the memory resources allocated for the request. */
aos_pool_destroy(pool);
/* Release the allocated global resources. */
aos_http_io_deinitialize();
return 0;
}
require 'aliyun/oss'
client = Aliyun::OSS::Client.new(
# In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint.
endpoint: 'https://oss-cn-hangzhou.aliyuncs.com',
# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
access_key_id: ENV['OSS_ACCESS_KEY_ID'],
access_key_secret: ENV['OSS_ACCESS_KEY_SECRET']
)
# Specify the name of the bucket. Example: examplebucket.
bucket = client.get_bucket('examplebucket')
# Set logging_bucket to the destination bucket in which log objects are stored.
# Set my-log to the directory in which log objects are stored. If you specify this parameter, log objects are stored in the specified directory of the destination bucket. If you do not specify this parameter, log objects are stored in the root directory of the destination bucket.
bucket.logging = Aliyun::OSS::BucketLogging.new(
enable: true, target_bucket: 'logging_bucket', target_prefix: 'my-log')
Use ossutil
You can use ossutil to configure logging. For more information, see Enable logging.
Use the OSS API
If your business requires a high level of customization, you can directly call RESTful APIs. To directly call an API, you must include the signature calculation in your code. For more information, see PutBucketLogging.
FAQ
Can I query interrupted requests in OSS access logs?
No, you cannot query interrupted requests in OSS access logs. OSS does not record interrupted requests in access logs. If you send a request by using an OSS SDK, you can identify the causes of request interruption based on the value returned by the OSS SDK.