Object Storage Service:Perform multipart upload by using OSS SDK for Go

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.

• In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

• To perform multipart upload, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.

• OSS SDK for Go 2.2.5 and later support all attributes that are included in the following sample code.

Process

To upload a local file by using multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Call the Bucket.InitiateMultipartUpload method to obtain a unique upload ID in OSS.

2. Upload parts.

   Call the Bucket.UploadPart method to upload the parts.

   Note

   • For parts that are uploaded by running a multipart upload task with a specific upload ID, the part numbers identify their relative positions in an object. If you upload a part and reuse its part number to upload another part, the new part overwrites the original part.

   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.

   • OSS calculates the MD5 hash of uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Go. If the two hashes are different, OSS returns the InvalidDigest error code.

3. Complete the multipart upload task.

   After you upload all parts, call the Bucket.CompleteMultipartUpload method to combine the parts into a complete object.

Examples

The following sample code provides an example on how to run a multipart upload task by following the multipart upload process:

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/aliyun/aliyun-oss-go-sdk/oss"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		log.Fatalf("Error: %v", err)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	// Specify the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the region to cn-hangzhou.
	clientOptions := []oss.ClientOption{oss.SetCredentialsProvider(&provider)}
	clientOptions = append(clientOptions, oss.Region("yourRegion"))
	// Specify the version of the signature algorithm.
	clientOptions = append(clientOptions, oss.AuthVersion(oss.AuthV4))
	client, err := oss.New("yourEndpoint", "", "", clientOptions...)
	if err != nil {
		log.Fatalf("Error: %v", err)
	}

	// Specify the name of the bucket.
	bucketName := "examplebucket"
	// Specify the full path of the object. Do not include the bucket name in the full path. 
	objectName := "exampleobject.txt"
	// Specify the full path of the local file. By default, if you do not specify the path of the local file, the file is uploaded from the path of the project to which the sample program belongs. 
	localFilename := "/localpath/exampleobject.txt"

	bucket, err := client.Bucket(bucketName)
	if err != nil {
		log.Fatalf("Error: %v", err)
	}

        // Specify the part size. Unit: bytes. In this example, the part size is set to 5 MB. 
	partSize := int64(5 * 1024 * 1024)

	// Call the multipart upload function. 
	if err := uploadMultipart(bucket, objectName, localFilename, partSize); err != nil {
		log.Fatalf("Failed to upload multipart: %v", err)
	}

}

// Specify the multipart upload function.
func uploadMultipart(bucket *oss.Bucket, objectName, localFilename string, partSize int64) error {
	// Split the local file.
	chunks, err := oss.SplitFileByPartSize(localFilename, partSize)
	if err != nil {
		return fmt.Errorf("failed to split file into chunks: %w", err)
	}

	// Open the local file. 
	file, err := os.Open(localFilename)
	if err != nil {
		return fmt.Errorf("failed to open file: %w", err)
	}
	defer file.Close()

	// Step 1: Initiate a multipart upload task. 
	imur, err := bucket.InitiateMultipartUpload(objectName)
	if err != nil {
		return fmt.Errorf("failed to initiate multipart upload: %w", err)
	}

	// Step 2: Upload the parts. 
	var parts []oss.UploadPart
	for _, chunk := range chunks {
		part, err := bucket.UploadPart(imur, file, chunk.Size, chunk.Number)
		if err != nil {
			// If a part fails to be uploaded, cancel the multipart upload task. 
			if abortErr := bucket.AbortMultipartUpload(imur); abortErr != nil {
				log.Printf("Failed to abort multipart upload: %v", abortErr)
			}
			return fmt.Errorf("failed to upload part: %w", err)
		}
		parts = append(parts, part)
	}

	// Set the access control list (ACL) of the object to private. By default, the object inherits the ACL of the bucket. 
	objectAcl := oss.ObjectACL(oss.ACLPrivate)

	// Step 3: Complete the multipart upload task. 
	_, err = bucket.CompleteMultipartUpload(imur, parts, objectAcl)
	if err != nil {
		// If you fail to complete the multipart upload task, cancel the multipart upload task. 
		if abortErr := bucket.AbortMultipartUpload(imur); abortErr != nil {
			log.Printf("Failed to abort multipart upload: %v", abortErr)
		}
		return fmt.Errorf("failed to complete multipart upload: %w", err)
	}

	log.Printf("Multipart upload completed successfully.")
	return nil
}

FAQ

...
if err = bucket.AbortMultipartUpload(imur); err != nil {
log.Fatalf("failed to abort multipart upload: %w", err)
}

log.Printf("Multipart upload aborted successfully.")

...	
if lsRes, err := bucket.ListUploadedParts(imur); err != nil {
log.Fatalf("Failed to list uploaded parts: %v", err)
}

for _, upload := range lsRes.UploadedParts {
log.Printf("List PartNumber: %d, ETag: %s, LastModified: %v\n", upload.PartNumber, upload.ETag, upload.LastModified)
}

References

• For the complete sample code that is used to perform multipart upload, visit GitHub.

• A multipart upload involves three API operations. For more information about the operations, see the following topics:

  • InitiateMultipartUpload

  • UploadPart

  • CompleteMultipartUpload

• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.

• For more information about the API operation that you can call to list uploaded parts, see ListUploadedParts.

• For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that were initiated but are not completed or canceled.