Object Storage Service:Multipart upload

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions, endpoints and open ports.

• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

• To perform multipart upload, you must have the oss:PutObject permission. For more information, see Attach a custom policy to a RAM user.

Procedure

To upload an object by using multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Call the InitiateMultipartUploadRequest method to obtain a unique upload ID in OSS.

2. Upload parts.

   Call the UploadPartRequest method to upload parts.

   Note

   • In a multipart upload task, part numbers are used to identify the relative positions of the parts in an object. If you upload a part that has the same part number as an existing part, the existing part is overwritten by the uploaded part.

   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.

   • OSS calculates the MD5 hash of uploaded parts and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Java. If the two hashes are different, OSS returns the InvalidDigest error code.

3. Complete the multipart upload task.

   After all parts are uploaded, call the CompleteMultipartUploadRequest method to combine the parts into a complete object.

Sample code of multipart upload

The following sample code provides an example on how to implement a multipart upload task by following the multipart upload process:

using Aliyun.OSS;

// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
var endpoint = "yourEndpoint";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
// Specify the name of the bucket. 
var bucketName = "examplebucket";
// Specify the full path of the object. Do not include the bucket name in the full path. 
var objectName = "exampleobject.txt";
// Specify the full path of the local file that you want to upload. By default, if you do not specify the full path of a local file, the local file is uploaded from the path of the project to which the sample program belongs. 
var localFilename = "D:\\localpath\\examplefile.txt";
// Create an OSSClient instance. 
var client = new OssClient(endpoint, accessKeyId, accessKeySecret);

// Initiate the multipart upload task and obtain the upload ID in the response. 
var uploadId = "";
try
{
    // Specify the name of the uploaded object and the bucket for the object. You can configure object metadata in InitiateMultipartUploadRequest. However, you do not need to specify ContentLength. 
    var request = new InitiateMultipartUploadRequest(bucketName, objectName);
    var result = client.InitiateMultipartUpload(request);
    uploadId = result.UploadId;
    // Display the upload ID. 
    Console.WriteLine("Init multi part upload succeeded");
    // Cancel the multipart upload task or list uploaded parts based on the upload ID. 
    // If you want to cancel a multipart upload task based on the upload ID, obtain the upload ID after you call the InitiateMultipartUpload operation to initiate the multipart upload task.  
    // If you want to list the uploaded parts in a multipart upload task based on the upload ID, obtain the upload ID after you call the InitiateMultipartUpload operation to initiate the multipart upload task but before you call the CompleteMultipartUpload operation to complete the multipart upload task. 
    Console.WriteLine("Upload Id:{0}", result.UploadId);
}
catch (Exception ex)
{
    Console.WriteLine("Init multi part upload failed, {0}", ex.Message);
    Environment.Exit(1);
}
// Calculate the total number of parts. 
var partSize = 100 * 1024;
var fi = new FileInfo(localFilename);
var fileSize = fi.Length;
var partCount = fileSize / partSize;
if (fileSize % partSize != 0)
{
    partCount++;
}
// Initialize parts and start the multipart upload task. partETags is a list of PartETags. After OSS receives the partETags, OSS verifies all parts one by one. After all parts are verified, OSS combines these parts into a complete object. 
var partETags = new List<PartETag>();
try
{
    using (var fs = File.Open(localFilename, FileMode.Open))
    {
        for (var i = 0; i < partCount; i++)
        {
            var skipBytes = (long)partSize * i;
            // Find the start position of the current upload task. 
            fs.Seek(skipBytes, 0);
            // Calculate the part size in this upload. The size of the last part is the size of the remainder after the object is split by the calculated part size. 
            var size = (partSize < fileSize - skipBytes) ? partSize : (fileSize - skipBytes);
            var request = new UploadPartRequest(bucketName, objectName, uploadId)
            {
                InputStream = fs,
                PartSize = size,
                PartNumber = i + 1
            };
            // Call UploadPart to upload parts. The returned results contain the ETag values of parts. 
            var result = client.UploadPart(request);
            partETags.Add(result.PartETag);
            Console.WriteLine("finish {0}/{1}", partETags.Count, partCount);
        }
        Console.WriteLine("Put multi part upload succeeded");
    }
}
catch (Exception ex)
{
    Console.WriteLine("Put multi part upload failed, {0}", ex.Message);
    Environment.Exit(1);
}
// Combine the parts after the parts are uploaded. 
try
{
    var completeMultipartUploadRequest = new CompleteMultipartUploadRequest(bucketName, objectName, uploadId);
    foreach (var partETag in partETags)
    {
        completeMultipartUploadRequest.PartETags.Add(partETag);
    }
    var result = client.CompleteMultipartUpload(completeMultipartUploadRequest);
    Console.WriteLine("complete multi part succeeded");
}
catch (Exception ex)
{
    Console.WriteLine("complete multi part failed, {0}", ex.Message);
    Environment.Exit(1);
}

Cancel a multipart upload task

You can call the client.AbortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to upload any parts. The uploaded parts are deleted.

The following sample code provides an example on how to cancel a multipart upload task:

using Aliyun.OSS;

// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
var endpoint = "yourEndpoint";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
// Specify the name of the bucket. 
var bucketName = "examplebucket";
// Specify the full path of the object. Do not include the bucket name in the full path. 
var objectName = "exampleobject.txt";
// Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. 
var uploadId = "yourUploadId";

// Create an OSSClient instance. 
var client = new OssClient(endpoint, accessKeyId, accessKeySecret);
// Initiate the multipart upload task. 
try
{
    var request = new InitiateMultipartUploadRequest(bucketName, objectName);
    var result = client.InitiateMultipartUpload(request);
    uploadId = result.UploadId;
    // Display the upload ID. 
    Console.WriteLine("Init multi part upload succeeded");
    Console.WriteLine("Upload Id:{0}", result.UploadId);
}
catch (Exception ex)
{
    Console.WriteLine("Init multi part upload failed, {0}", ex.Message);
}
// Cancel the multipart upload task. 
try
{
    var request = new AbortMultipartUploadRequest(bucketName, objectName, uploadId);
    client.AbortMultipartUpload(request);
    Console.WriteLine("Abort multi part succeeded, {0}", uploadId);
}
catch (Exception ex)
{
    Console.WriteLine("Abort multi part failed, {0}", ex.Message);
}

List uploaded parts

The following code provides an example on how to list uploaded parts:

using Aliyun.OSS;
// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
var endpoint = "yourEndpoint";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
// Specify the name of the bucket. 
var bucketName = "examplebucket";
// Specify the full path of the object. Do not include the bucket name in the full path. 
var objectName = "exampleobject.txt";
// Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. You must obtain the upload ID before you call the CompleteMultipartUpload operation to complete the multipart upload task. 
var uploadId = "yourUploadId";

// Create an OSSClient instance. 
var client = new OssClient(endpoint, accessKeyId, accessKeySecret);
try
{
    PartListing listPartsResult = null;
    var nextMarker = 0;
    do 
    {
        var listPartsRequest = new ListPartsRequest(bucketName, objectName, uploadId) 
        {
            PartNumberMarker = nextMarker,
        };
        // List the uploaded parts. 
        listPartsResult = client.ListParts(listPartsRequest);
        Console.WriteLine("List parts succeeded");
        // Traverse all the uploaded parts. 
        var parts = listPartsResult.Parts;
        foreach (var part in parts)
        {
            Console.WriteLine("partNumber: {0}, ETag: {1}, Size: {2}", part.PartNumber, part.ETag, part.Size);
        }
        nextMarker = listPartsResult.NextPartNumberMarker;
    } while (listPartsResult.IsTruncated);
}
catch (Exception ex)
{
    Console.WriteLine("List parts failed, {0}", ex.Message);
}

List multipart upload tasks

You can call the ossClient.listMultipartUploads method to list all ongoing multipart upload tasks, which are tasks that have been initiated but are not completed or canceled. The following table describes the parameters that you can configure when you list all ongoing multipart upload tasks.

The following sample code provides an example on how to use simple list to list multipart upload tasks:

using Aliyun.OSS;
// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
var endpoint = "yourEndpoint";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
// Specify the name of the bucket. 
var bucketName = "examplebucket";

// Create an OSSClient instance. 
var client = new OssClient(endpoint, accessKeyId, accessKeySecret);
try
{
    MultipartUploadListing multipartUploadListing = null;
    var nextMarker = string.Empty;
    do
    {
        // List multipart upload tasks. 
        var request = new ListMultipartUploadsRequest(bucketName)
        {
            KeyMarker = nextMarker,
        };
        multipartUploadListing = client.ListMultipartUploads(request);
        Console.WriteLine("List multi part succeeded");
        // List information about each multipart upload task. 
        foreach (var mu in multipartUploadListing.MultipartUploads)
        {
            Console.WriteLine("Key: {0},  UploadId: {1}", mu.Key, mu.UploadId);
        }
        // If the value of the isTruncated field in the returned result is false, values of nextKeyMarker and nextUploadIdMarker are returned and used as the start point for the next read. 
        nextMarker = multipartUploadListing.NextKeyMarker;
    } while (multipartUploadListing.IsTruncated);
}
catch (Exception ex)
{
    Console.WriteLine("List multi part uploads failed, {0}", ex.Message);
}

List multipart upload tasks by specifying a prefix and the maximum number of records to return

The following code provides an example on how to list multipart upload tasks by specifying a prefix and the maximum number of records to return:

using Aliyun.OSS;
// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
var endpoint = "yourEndpoint";
// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
var accessKeyId = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_ID");
var accessKeySecret = Environment.GetEnvironmentVariable("OSS_ACCESS_KEY_SECRET");
// Specify the name of the bucket. 
var bucketName = "examplebucket";

// Specify the prefix. 
var prefix = "yourObjectPrefix";
// Specify the maximum number of records to return as 100. 
var maxUploads = 100;
// Create an OSSClient instance. 
var client = new OssClient(endpoint, accessKeyId, accessKeySecret);
try
{
    MultipartUploadListing multipartUploadListing = null;
    var nextMarker = string.Empty;
    do
    {
        // List multipart upload tasks. By default, 1,000 multipart upload tasks are listed. 
        var request = new ListMultipartUploadsRequest(bucketName)
        {
            KeyMarker = nextMarker,
            // Specify the prefix that is contained in the names of the objects that you want to list. 
            Prefix = prefix,
            // Specify the maximum number of records to return. 
            MaxUploads = maxUploads,
        };
        multipartUploadListing = client.ListMultipartUploads(request);
        Console.WriteLine("List multi part succeeded");
        // List information about each multipart upload task. 
        foreach (var mu in multipartUploadListing.MultipartUploads)
        {
            Console.WriteLine("Key: {0},  UploadId: {1}", mu.Key, mu.UploadId);
        }
        nextMarker = multipartUploadListing.NextKeyMarker;
    } while (multipartUploadListing.IsTruncated);
}
catch (Exception ex)
{
    Console.WriteLine("List multi part uploads failed, {0}", ex.Message);
}

References

• For the complete sample code that is used to perform multipart upload, visit GitHub.

• A multipart upload involves three API operations. For more information about the operations, see the following topics:

  • InitiateMultipartUpload

  • UploadPart

  • CompleteMultipartUpload

• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.

• For more information about the API operation that you can call to list uploaded parts, see ListUploadedParts.

• For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks include tasks that were initiated but are not completed or canceled.