To improve the efficiency of uploading a large object, Object Storage Service (OSS) provides two methods: resumable upload and multipart upload. This topic describes the preceding upload methods and provides links to the related documents for further reference.
Usage notes
If you use simple upload, such as PutObject, to upload a large object, the following issues may occur:
Limit on the maximum size of object : You cannot upload an object larger than 5 GB in size by calling the
PutObjectoperation. Attempts to upload an object larger than 5 GB in size fail.Amount of time required for object upload: A long period of time is required to upload a large object, especially when the network connection is poor. Long-term data transmission is vulnerable to network fluctuations, which may cause transmission interruption or timeout issues.
Memory consumption: Uploading a large object may increase memory consumption, especially when the memory is small or the environment is limited, which affects the system stability and performance.
Therefore, we recommend that you use resumable upload or multipart upload to improve efficiency and reliability when you upload a large object.
Resumable upload
Resumable upload is an efficient method to upload large objects, especially when the network connection is unstable or the object size exceeds 5 GB. Resumable upload automatically splits an object into multiple parts and uploads the parts in parallel. During resumable upload, the upload progress is recorded in a checkpoint file. If the object upload is interrupted, the object is re-uploaded from the position that is recorded in the checkpoint file. You do not need to re-upload the whole object, which significantly improves the upload efficiency and reliability.
Multipart upload
Multipart upload is a common method to upload large objects. You must manually split a large object into multiple parts and then call the multipart upload operations to upload the parts. Compared with resumable upload, multipart upload provides higher flexibility and allows you to dynamically adjust the size of parts and cancel multipart upload tasks during the upload process. Multipart upload is suitable for scenarios in which fine-grained management of the upload process is required.
Method comparison
Item | Resumable upload | Multipart upload |
Scenario | Users want to upload large objects in a simple and fast manner and do not want to focus on specific details. | Users require fine-grained management of the upload process and have technical capabilities. |
Concurrent upload | Supported. | Supported. |
Object re-upload from the failed position | Supported. You can re-upload the object from the position that is recorded in the checkpoint file. | Not supported. You cannot directly re-upload the object from the position that is recorded in the checkpoint file. |
Part size | You must preset the size of parts. | You can dynamically adjust the size of parts. |
Reliability | High. Resumable upload can reduce upload failures due to network fluctuations. | Lower than resumable upload. You must manually resolve the failed parts. |
Management difficulty | Low. You need to only call the UploadFile operation. | High. You need to call the InitiateMultipartUpload, UploadPart, and CompleteMultipartUpload operations. |
Flexibility | You cannot manage the upload process in a fine-grained manner. | High. You can manually manage the upload process to facilitate real-time monitoring of multipart upload tasks. |
Development difficulty | Extremely low. You need to only call the UploadFile operation to complete the resumable upload task. | High. You need to manually split the large object, write more management logic to call multiple operations to complete the multipart upload task, and handle various exceptions. |