Object Storage Service:Append upload

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions, endpoints and open ports.

• In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

• To use append upload, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.

• If the object to which you want to append content does not exist, the AppendObject operation creates an appendable object.

• If the object to which you want to append content exists:

  • If the object is an appendable object and the specified position from which the append operation starts is equal to the current object size, the data is appended to the end of the object.

  • If the object is an appendable object and the specified position from which the append operation starts is not equal to the current object size, the PositionNotEqualToLength error is returned.

  • If the object is not an appendable object, the ObjectNotAppendable error is returned.

Sample code

The following sample code provides an example on how to perform append upload:

package main

import (
	"log"
	"strings"
	"time"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Failed to create credentials provider: %v", err)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint.
	// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou. Specify the actual region.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Specify the version of the signature algorithm.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Failed to create OSS client: %v", err)
	}

	// Specify the name of the bucket. Example: examplebucket. 
	bucketName := "examplebucket" // Replace it with the actual name of the bucket.
	bucket, err := client.Bucket(bucketName)
	if err != nil {
		log.Fatalf("Failed to get bucket: %v", err)
	}

	// Specify the full path of the object. Do not include the bucket name in the full path. Example: appendobject.txt. 
	objectName := "appendobject.txt"
	var nextPos int64 = 0

	// Specify the expiration time of the request. 
	expires := time.Date(2025, time.December, 10, 23, 0, 0, 0, time.UTC)
	options := []oss.Option{
		oss.Expires(expires),
	}

	// If the object is appended for the first time, the position from which the append operation starts is 0. The position for the next append operation is included in the response.
	appendValue1 := "YourObjectAppendValue1"
	nextPos, err = bucket.AppendObject(objectName, strings.NewReader(appendValue1), nextPos, options...)
	if err != nil {
		log.Fatalf("Failed to append first value: %v", err)
	}

	// Perform the second append operation.
	appendValue2 := "YourObjectAppendValue2"
	nextPos, err = bucket.AppendObject(objectName, strings.NewReader(appendValue2), nextPos)
	if err != nil {
		log.Fatalf("Failed to append second value: %v", err)
	}

	log.Println("Append uploads completed successfully.")
}

FAQ

How to use AppendObject if you have appended content to the object before?

In the following scenarios, to make sure that the object is appended to the correct position, you can use the GetObjectDetailedMeta method to obtain metadata of the object, including the position where the last append operation ends.

1. Collection of application logs:

• A large amount of logs is generated by running applications. Append upload helps you save the effort of creating new objects each time, for you can append log data to the same object continuously by means of append upload.

2. Real-time data analysis:

• Real-time data analysis systems typically send continuous streams of data to storage systems. By using append upload, you can add these data streams to the same object in real time, which is convenient for subsequent batch or real-time processing.

...
props, err := bucket.GetObjectDetailedMeta(objectName)
if err != nil {
	log.Fatalf("Failed to get object detailed meta: %v", err)
}
nextPos, err = strconv.ParseInt(props.Get("X-Oss-Next-Append-Position"), 10, 64)
if err != nil {
	log.Fatalf("Failed to parse next position: %v", err)
}

// Perform append operation.
appendValue2 := "YourObjectAppendValue2"
nextPos, err = bucket.AppendObject(objectName, strings.NewReader(appendValue2), nextPos)
if err != nil {
	log.Fatalf("Failed to append second value: %v", err)
}

log.Println("Append uploads completed successfully.")

References

• For the complete sample code that is used to perform append upload, visit GitHub.

• For more information about the API operation that you can call to perform append upload, see AppendObject.