Object Storage Service:Multipart upload

Usage notes

• Before you run the sample code in this topic, you must create an OSSClient instance by using methods such as using a custom domain name or Security Token Service (STS). For more information, see Initialization.

Process

To upload an object by using multipart upload, perform the following steps:

1. Initiate a multipart upload task.

   Call the oss.initMultipartUpload method to create a unique upload ID in OSS.

2. Upload parts.

   Call the oss.uploadPart method to upload parts.

   Note

   • If parts are uploaded by a multipart upload task that has a specific upload ID, part numbers are used to identify the relative positions of the parts in an object. If you upload a part that has the same part number as an existing part, the existing part is overwritten by the uploaded part.

   • OSS includes the MD5 hash of each uploaded part in the ETag header in the response.

   • OSS calculates the MD5 hash of uploaded data and compares the MD5 hash with the MD5 hash that is calculated by OSS SDK for Java. If the two hashes are different, OSS returns the InvalidDigest error code.

3. Complete the multipart upload task.

   After all parts are uploaded, call the oss.CompleteMultipartUpload method to combine the parts into a complete object.

Complete sample code

The following sample code provides an example on how to implement a multipart upload task by following the multipart upload process:

// Specify the name of the bucket. Example: examplebucket. 
String bucketName = "examplebucket";
// Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
String objectName = "exampledir/exampleobject.txt";
// Specify the full path of the local file. Example: /storage/emulated/0/oss/examplefile.txt. 
String localFilepath = "/storage/emulated/0/oss/examplefile.txt";

// Initiate a multipart upload task. 
InitiateMultipartUploadRequest init = new InitiateMultipartUploadRequest(bucketName, objectName);
InitiateMultipartUploadResult initResult = oss.initMultipartUpload(init);
// Obtain the upload ID. 
String uploadId = initResult.getUploadId();
// Cancel the multipart upload task or list uploaded parts based on the upload ID. 
// If you want to cancel a multipart upload task based on the upload ID, obtain the upload ID after you call the InitiateMultipartUpload operation to initiate the multipart upload task.  
// If you want to list the uploaded parts in a multipart upload task based on the upload ID, obtain the upload ID after you call the InitiateMultipartUpload operation to initiate the multipart upload task and before you call the CompleteMultipartUpload operation to complete the multipart upload task. 
// Log.d("uploadId", uploadId);

// Specify the part size. Unit: bytes. Valid values: 100 KB to 5 GB. 
int partCount = 100 * 1024;
// Start the multipart upload task. 
List<PartETag> partETags = new ArrayList<>();
for (int i = 1; i < 5; i++) {
    byte[] data = new byte[partCount];

    RandomAccessFile raf = new RandomAccessFile(localFilepath, "r");
    long skip = (i-1) * partCount;
    raf.seek(skip);
    raf.readFully(data, 0, partCount);

    UploadPartRequest uploadPart = new UploadPartRequest();
    uploadPart.setBucketName(bucketName);
    uploadPart.setObjectKey(objectName);
    uploadPart.setUploadId(uploadId);
    // Specify the part number of each part. The number starts from 1. Each part has a part number. Valid values: 1 to 10000. 
    uploadPart.setPartNumber(i); 
    uploadPart.setPartContent(data);
    try {
        UploadPartResult result = oss.uploadPart(uploadPart);
        PartETag partETag = new PartETag(uploadPart.getPartNumber(), result.getETag());
        partETags.add(partETag);
    } catch (ServiceException serviceException) {
        OSSLog.logError(serviceException.getErrorCode());
    }
}
Collections.sort(partETags, new Comparator<PartETag>() {
    @Override
    public int compare(PartETag lhs, PartETag rhs) {
        if (lhs.getPartNumber() < rhs.getPartNumber()) {
            return -1;
        } else if (lhs.getPartNumber() > rhs.getPartNumber()) {
            return 1;
        } else {
            return 0;
        }
    }
});

// Complete the multipart upload task. 
CompleteMultipartUploadRequest complete = new CompleteMultipartUploadRequest(bucketName, objectName, uploadId, partETags);

// Implement upload callback. You can configure the CALLBACK_SERVER parameter when you complete the multipart upload task. A callback request is sent to the specified server address after you complete the multipart upload task. You can view the servercallback result in completeResult.getServerCallbackReturnBody() of the response. 
complete.setCallbackParam(new HashMap<String, String>() {
    {
        put("callbackUrl", CALLBACK_SERVER); // Set the CALLBACK_SERVER parameter to your server address. 
        put("callbackBody", "test");
    }
});
CompleteMultipartUploadResult completeResult = oss.completeMultipartUpload(complete);
OSSLog.logError("-------------- serverCallback: " + completeResult.getServerCallbackReturnBody());

The preceding code uses uploadPart to upload each part.

• You must specify the upload ID and part numbers for each multipart upload task. Valid values of PartNumber range from 1 to 10000. If a part number is not within this range, the InvalidArgument error code is returned.

• If you use uploadPart, parts must be larger than 100 KB except for the last part. When you use uploadPart, the size of each part is verified only after all parts are uploaded.

• Each time you upload a part, make sure that the stream is directed to the start position of the part.

• Each time a part is uploaded, OSS returns a response that contains the ETag value of the part. If the ETag value is the same as the MD5 hash, combine the ETag value with the part number into a PartETag and save the PartETag for subsequent uploads of parts.

Upload a local file by using multipart upload

You can upload local files to OSS by using multipart upload in synchronous mode or in asynchronous mode.

• Upload a local file by using multipart upload in synchronous mode

  The following sample code provides an example on how to upload a local file named examplefile.txt to the exampleobject.txt object in the exampledir directory of the examplebucket bucket by using multipart upload in synchronous mode:

  // Specify the name of the bucket. Example: examplebucket. 
  String bucketName = "examplebucket";
  // Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
  String objectName = "exampledir/exampleobject.txt";
  // Specify the full path of the local file. Example: /storage/emulated/0/oss/examplefile.txt. 
  String localFilepath = "/storage/emulated/0/oss/examplefile.txt";
  
  ObjectMetadata meta = new ObjectMetadata();
  // Configure object metadata. 
  meta.setHeader("x-oss-object-acl", "public-read-write");
  MultipartUploadRequest rq = new MultipartUploadRequest(bucketName, objectName, localFilepath, meta);
  // Specify the part size. The default value of PartSize is 256 KB. The minimum value is 100 KB. 
  rq.setPartSize(1024 * 1024);
  rq.setProgressCallback(new OSSProgressCallback<MultipartUploadRequest>() {
      @Override
      public void onProgress(MultipartUploadRequest request, long currentSize, long totalSize) {
          OSSLog.logDebug("[testMultipartUpload] - " + currentSize + " " + totalSize, false);
      }
  });
  
  CompleteMultipartUploadResult result = oss.multipartUpload(rq);

  For scoped storage on Android 10 or later, you can use the Uniform Resource Identifier (URI) of a file to upload the file to OSS.

  // Specify the name of the bucket. Example: examplebucket. 
  String bucketName = "examplebucket";
  // Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
  String objectName = "exampledir/exampleobject.txt";
  
  ObjectMetadata meta = new ObjectMetadata();
  // Configure object metadata. 
  meta.setHeader("x-oss-object-acl", "public-read-write");
  MultipartUploadRequest rq = new MultipartUploadRequest(bucketName, objectName, fileUri, meta);
  // Specify the part size. The default value of PartSize is 256 KB. The minimum value is 100 KB. 
  rq.setPartSize(1024 * 1024);
  rq.setProgressCallback(new OSSProgressCallback<MultipartUploadRequest>() {
      @Override
      public void onProgress(MultipartUploadRequest request, long currentSize, long totalSize) {
          OSSLog.logDebug("[testMultipartUpload] - " + currentSize + " " + totalSize, false);
      }
  });
  
  CompleteMultipartUploadResult result = oss.multipartUpload(rq);

• Upload a local file by using multipart upload in asynchronous mode

  The following sample code provides an example on how to upload a local file named examplefile.txt to the exampleobject.txt object in the exampledir directory of the examplebucket bucket by using multipart upload in asynchronous mode:

  // Specify the name of the bucket. Example: examplebucket. 
  String bucketName = "examplebucket";
  // Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
  String objectName = "exampledir/exampleobject.txt";
  // Specify the full path of the local file. Example: /storage/emulated/0/oss/examplefile.txt. 
  String localFilepath = "/storage/emulated/0/oss/examplefile.txt";
  
  MultipartUploadRequest request = new MultipartUploadRequest(bucketName, objectName, localFilepath);
  
  request.setProgressCallback(new OSSProgressCallback<MultipartUploadRequest>() {
      @Override
      public void onProgress(MultipartUploadRequest request, long currentSize, long totalSize) {
          OSSLog.logDebug("[testMultipartUpload] - " + currentSize + " " + totalSize, false);
      }
  });
  
  OSSAsyncTask task = oss.asyncMultipartUpload(request, new OSSCompletedCallback<MultipartUploadRequest, CompleteMultipartUploadResult>() {
      @Override
      public void onSuccess(MultipartUploadRequest request, CompleteMultipartUploadResult result) {
          OSSLog.logInfo(result.getServerCallbackReturnBody());
      }
  
      @Override
      public void onFailure(MultipartUploadRequest request, ClientException clientException, ServiceException serviceException) {
          OSSLog.logError(serviceException.getRawMessage());
      }
  });
  
  //Thread.sleep(100);
  // Cancel the multipart upload task. 
  //task.cancel();   
  
  task.waitUntilFinished();

  For scoped storage on Android 10 or later, you can use the URI of a file to upload the file to OSS.

  // Specify the name of the bucket. Example: examplebucket. 
  String bucketName = "examplebucket";
  // Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
  String objectName = "exampledir/exampleobject.txt";
  
  MultipartUploadRequest request = new MultipartUploadRequest(bucketName, objectName, fileUri);
  
  request.setProgressCallback(new OSSProgressCallback<MultipartUploadRequest>() {
      @Override
      public void onProgress(MultipartUploadRequest request, long currentSize, long totalSize) {
          OSSLog.logDebug("[testMultipartUpload] - " + currentSize + " " + totalSize, false);
      }
  });
  
  OSSAsyncTask task = oss.asyncMultipartUpload(request, new OSSCompletedCallback<MultipartUploadRequest, CompleteMultipartUploadResult>() {
      @Override
      public void onSuccess(MultipartUploadRequest request, CompleteMultipartUploadResult result) {
          OSSLog.logInfo(result.getServerCallbackReturnBody());
      }
  
      @Override
      public void onFailure(MultipartUploadRequest request, ClientException clientException, ServiceException serviceException) {
          OSSLog.logError(serviceException.getRawMessage());
      }
  });
  
  //Thread.sleep(100);
  // Cancel the multipart upload task. 
  //task.cancel();   
  
  task.waitUntilFinished();

List uploaded parts

You can call the oss.listParts method to obtain all parts that are uploaded in a multipart upload task.

The following sample code provides an example on how to list uploaded parts:

// Specify the name of the bucket. Example: examplebucket. 
String bucketName = "examplebucket";
// Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
String objectName = "exampledir/exampleobject.txt";
// Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. You must obtain the upload ID before you call the CompleteMultipartUpload operation to complete the multipart upload task. 
String uploadId = "0004B999EF518A1FE585B0C9****";

// List uploaded parts. 
ListPartsRequest listParts = new ListPartsRequest(bucketName, objectName, uploadId);
ListPartsResult result = oss.listParts(listParts);

List<PartETag> partETagList = new ArrayList<PartETag>();
for (PartSummary part : result.getParts()) {
    partETagList.add(new PartETag(part.getPartNumber(), part.getETag()));
}

Cancel a multipart upload task

You can call the oss.abortMultipartUpload method to cancel a multipart upload task. If you cancel a multipart upload task, you cannot use the upload ID to perform multipart operations. The uploaded parts are deleted.

The following sample code provides an example on how to cancel a multipart upload task:

// Specify the name of the bucket. Example: examplebucket. 
String bucketName = "examplebucket";
// Specify the full path of the object. Example: exampledir/exampleobject.txt. Do not include the bucket name in the full path. 
String objectName = "exampledir/exampleobject.txt";
// Specify the upload ID. You can obtain the upload ID from the response to the InitiateMultipartUpload operation. 
String uploadId = "0004B999EF518A1FE585B0C9****";

// Cancel the multipart upload task. 
AbortMultipartUploadRequest abort = new AbortMultipartUploadRequest(bucketName, objectName, uploadId);
AbortMultipartUploadResult abortResult = oss.abortMultipartUpload(abort);

References

• For the complete sample code that is used to perform multipart upload, visit GitHub.

• A multipart upload involves three API operations. For more information about the operations, see the following topics:

  • InitiateMultipartUpload

  • UploadPart

  • CompleteMultipartUpload

• For more information about the API operation that you can call to cancel a multipart upload task, see AbortMultipartUpload.

• For more information about the API operation that you can call to list uploaded parts, see ListParts.

• For more information about the API operation that you can call to list all ongoing multipart upload tasks, see ListMultipartUploads. Ongoing multipart upload tasks are tasks that have been initiated but are not completed or canceled.

• For more information about how to initialize an OSSClient instance, see Initialization.