Implementing OSS Multipart Download for Mobile Devices

In this article, we will discuss the implementation of resumable downloads on mobile terminals using Object Storage Service's multipart download feature.

11.11 The Biggest Deals of the Year. 40% OFF on selected cloud servers with a free 100 GB data transfer! Click here to learn more.

When a user watches a video using a video app on his/her mobile phone, the video app pre-fetches the content for a seamless user experience. Normally, the app will switch the network from mobile data to Wi-Fi during the download to help users reduce data consumption. When the download resumes, unfinished parts can be downloaded separately from the already downloaded parts, which saves time and traffic.

In this article, we will discuss the implementation of this function on mobile terminals using the Alibaba Cloud Object Storage Service (OSS) multipart download feature.

Technical Points

Support for Range headers is added in HTTP 1.1 to specify the scope for getting data. The formats of Range headers generally include:

Range: bytes=100- Start from the 101st byte until finished.
Range: Bytes=100-200 Specify the length from the beginning to the end, remember that Range is counted from 0, so this requires the server to start from the 101st byte to the 201st byte. This is generally used for sharding a particularly large file, such as video.
Range: Bytes=-100 If the range does not specify a starting position, it requires the server to transfer the last 100 bytes of content, rather than 100 bytes starting from byte 0.
Range: bytes=0-100, 200-300 Multiple ranges of content can be specified at the same time, which is not common.

In addition, when the multipart is resumed, it is necessary to verify whether the file on the server has changed. At this time, the If-Match header is used. If-Match corresponds to the value of Etag.

When the client initiates the request with Range and If-Match attached in the header, the OSS server will verify the Etag value in 'If-Match' after receiving the request. If it does not match, it will return a 412 precondition status code.

The OSS server supports the Range, If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since for the open API getObject, so we can implement the multipart download function of OSS resources on the mobile terminal.

Best Practices

First let's look at the flow chart:

Our goal is to implement "pausable" or "resumable" downloads by using OSS multipart download feature.

Here is an example of iOS download implementation. The reference code is as follows. It is for reference only, and cannot be used for production.

#import "DownloadService.h"
#import "OSSTestMacros.h"

@implementation DownloadRequest

@end

@implementation Checkpoint

- (instancetype)copyWithZone:(NSZone *)zone {
    Checkpoint *other = [[[self class] allocWithZone:zone] init];
    
    other.etag = self.etag;
    other.totalExpectedLength = self.totalExpectedLength;
    
    return other;
}

@end


@interface DownloadService()<NSURLSessionTaskDelegate, NSURLSessionDataDelegate>

@property (nonatomic, strong) NSURLSession *session;         //network session
@property (nonatomic, strong) NSURLSessionDataTask *dataTask;   //data request task
@property (nonatomic, copy) DownloadFailureBlock failure;    //request failure
@property (nonatomic, copy) DownloadSuccessBlock success;    //request success
@property (nonatomic, copy) DownloadProgressBlock progress;  //download progress
@property (nonatomic, copy) Checkpoint *checkpoint;        //checkpoint
@property (nonatomic, copy) NSString *requestURLString;    //the URL of the file for download request
@property (nonatomic, copy) NSString *headURLString;       //the URL of the file for head request
@property (nonatomic, copy) NSString *targetPath;     //the path of the file
@property (nonatomic, assign) unsigned long long totalReceivedContentLength; //the length of the downloaded content
@property (nonatomic, strong) dispatch_semaphore_t semaphore;

@end

@implementation DownloadService

- (instancetype)init
{
    self = [super init];
    if (self) {
        NSURLSessionConfiguration *conf = [NSURLSessionConfiguration defaultSessionConfiguration];
        conf.timeoutIntervalForRequest = 15;
        
        NSOperationQueue *processQueue = [NSOperationQueue new];
        _session = [NSURLSession sessionWithConfiguration:conf delegate:self delegateQueue:processQueue];
        _semaphore = dispatch_semaphore_create(0);
        _checkpoint = [[Checkpoint alloc] init];
    }
    return self;
}

+ (instancetype)downloadServiceWithRequest:(DownloadRequest *)request {
    DownloadService *service = [[DownloadService alloc] init];
    if (service) {
        service.failure = request.failure;
        service.success = request.success;
        service.requestURLString = request.sourceURLString;
        service.headURLString = request.headURLString;
        service.targetPath = request.downloadFilePath;
        service.progress = request.downloadProgress;
        if (request.checkpoint) {
            service.checkpoint = request.checkpoint;
        }
    }
    return service;
}

/**
 * head file information, the etag of the extracted file is compared with the etag saved in the local checkpoint, and the result is returned
 */
- (BOOL)getFileInfo {
    __block BOOL resumable = NO;
    NSURL *url = [NSURL URLWithString:self.headURLString];
    NSMutableURLRequest *request = [[NSMutableURLRequest alloc]initWithURL:url];
    [request setHTTPMethod:@"HEAD"];
    
    NSURLSessionDataTask *task = [[NSURLSession sharedSession] dataTaskWithRequest:request completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
        if (error) {
            cNSLog(@"Failed to obtain file metadata, error : %@", error);
        } else {
            NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)response;
            NSString *etag = [httpResponse.allHeaderFields objectForKey:@"Etag"];
            if ([self.checkpoint.etag isEqualToString:etag]) {
                resumable = YES;
            } else {
                resumable = NO;
            }
        }
        dispatch_semaphore_signal(self.semaphore);
    }];
    [task resume];
    
    dispatch_semaphore_wait(self.semaphore, DISPATCH_TIME_FOREVER);
    return resumable;
}

/**
 * for obtaining the size of the local file
 */
- (unsigned long long)fileSizeAtPath:(NSString *)filePath {
    unsigned long long fileSize = 0;
    NSFileManager *dfm = [NSFileManager defaultManager];
    if ([dfm fileExistsAtPath:filePath]) {
        NSError *error = nil;
        NSDictionary *attributes = [dfm attributesOfItemAtPath:filePath error:&error];
        if (!error && attributes) {
            fileSize = attributes.fileSize;
        } else if (error) {
            NSLog(@"error: %@", error);
        }
    }

    return fileSize;
}

- (void)resume {
    NSURL *url = [NSURL URLWithString:self.requestURLString];
    NSMutableURLRequest *request = [[NSMutableURLRequest alloc]initWithURL:url];
    [request setHTTPMethod:@"GET"];
    
    BOOL resumable = [self getFileInfo];    // if the resumable is NO, the download cannot be resumed, otherwise go to the resume logic.
    if (resumable) {
        self.totalReceivedContentLength = [self fileSizeAtPath:self.targetPath];
        NSString *requestRange = [NSString stringWithFormat:@"bytes=%llu-", self.totalReceivedContentLength];
        [request setValue:requestRange forHTTPHeaderField:@"Range"];
    } else {
        self.totalReceivedContentLength = 0;
    }
    
    if (self.totalReceivedContentLength == 0) {
        [[NSFileManager defaultManager] createFileAtPath:self.targetPath contents:nil attributes:nil];
    }
    
    self.dataTask = [self.session dataTaskWithRequest:request];
    [self.dataTask resume];
}

- (void)pause {
    [self.dataTask cancel];
    self.dataTask = nil;
}

- (void)cancel {
    [self.dataTask cancel];
    self.dataTask = nil;
    [self removeFileAtPath: self.targetPath];
}

- (void)removeFileAtPath:(NSString *)filePath {
    NSError *error = nil;
    [[NSFileManager defaultManager] removeItemAtPath:self.targetPath error:&error];
    if (error) {
        NSLog(@"remove file with error : %@", error);
    }
}

#pragma mark - NSURLSessionDataDelegate

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task
didCompleteWithError:(nullable NSError *)error {
    NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)task.response;
    if ([httpResponse isKindOfClass:[NSHTTPURLResponse class]]) {
        if (httpResponse.statusCode == 200) {
            self.checkpoint.etag = [[httpResponse allHeaderFields] objectForKey:@"Etag"];
            self.checkpoint.totalExpectedLength = httpResponse.expectedContentLength;
        } else if (httpResponse.statusCode == 206) {
            self.checkpoint.etag = [[httpResponse allHeaderFields] objectForKey:@"Etag"];
            self.checkpoint.totalExpectedLength = self.totalReceivedContentLength + httpResponse.expectedContentLength;
        }
    }
    
    if (error) {
        if (self.failure) {
            NSMutableDictionary *userInfo = [NSMutableDictionary dictionaryWithDictionary:error.userInfo];
            [userInfo oss_setObject:self.checkpoint forKey:@"checkpoint"];
            
            NSError *tError = [NSError errorWithDomain:error.domain code:error.code userInfo:userInfo];
            self.failure(tError);
        }
    } else if (self.success) {
        self.success(@{@"status": @"success"});
    }
}

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveResponse:(NSURLResponse *)response completionHandler:(void (^)(NSURLSessionResponseDisposition))completionHandler
{
    NSHTTPURLResponse *httpResponse = (NSHTTPURLResponse *)dataTask.response;
    if ([httpResponse isKindOfClass:[NSHTTPURLResponse class]]) {
        if (httpResponse.statusCode == 200) {
            self.checkpoint.totalExpectedLength = httpResponse.expectedContentLength;
        } else if (httpResponse.statusCode == 206) {
            self.checkpoint.totalExpectedLength = self.totalReceivedContentLength +  httpResponse.expectedContentLength;
        }
    }
    
    completionHandler(NSURLSessionResponseAllow);
}

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveData:(NSData *)data {
    
    NSFileHandle *fileHandle = [NSFileHandle fileHandleForWritingAtPath:self.targetPath];
    [fileHandle seekToEndOfFile];
    [fileHandle writeData:data];
    [fileHandle closeFile];
    
    self.totalReceivedContentLength += data.length;
    if (self.progress) {
        self.progress(data.length, self.totalReceivedContentLength, self.checkpoint.totalExpectedLength);
    }
}

@end

Shown above is the processing logic for receiving data from the network, with DownloadService at the core of the download logic. In the URLSession:dataTask:didReceiveData, the received network data is appended to the file, and the download progress is updated. In the URLSession:task:didCompleteWithError, determine whether the download task is completed, and then return the result to the upper layer service. In the URLSession:dataTask:didReceiveResponse:completionHandler proxy method, the object-related information is used, for example etag for the multipart resume precheck, and content-length for calculating the download progress.

#import <Foundation/Foundation.h>

typedef void(^DownloadProgressBlock)(int64_t bytesReceived, int64_t totalBytesReceived, int64_t totalBytesExpectToReceived);
typedef void(^DownloadFailureBlock)(NSError *error);
typedef void(^DownloadSuccessBlock)(NSDictionary *result);

@interface Checkpoint : NSObject<NSCopying>

@property (nonatomic, copy) NSString *etag;     // etag value of the resource
@property (nonatomic, assign) unsigned long long totalExpectedLength;    //total length of the file

@end

@interface DownloadRequest : NSObject

@property (nonatomic, copy) NSString *sourceURLString;      // the URL for download

@property (nonatomic, copy) NSString *headURLString;        // the URL for obtaining the file source

@property (nonatomic, copy) NSString *downloadFilePath;     // the local path of the file

@property (nonatomic, copy) DownloadProgressBlock downloadProgress; // download progress

@property (nonatomic, copy) DownloadFailureBlock failure;   // callback for a download success

@property (nonatomic, copy) DownloadSuccessBlock success;   // callback for a download failure

@property (nonatomic, copy) Checkpoint *checkpoint;         // checkpoint for storing the etag of the file

@end


@interface DownloadService : NSObject

+ (instancetype)downloadServiceWithRequest:(DownloadRequest *)request;

/**
 * Start the download
 */
- (void)resume;

/**
 * Pause the download
 */
- (void)pause;

/**
 * Cancel the download
 */
- (void)cancel;

@end

The above section is the definition of DownloadRequest.

- (void)initDownloadURLs {
    OSSPlainTextAKSKPairCredentialProvider *pCredential = [[OSSPlainTextAKSKPairCredentialProvider alloc] initWithPlainTextAccessKey:OSS_ACCESSKEY_ID secretKey:OSS_SECRETKEY_ID];
    _mClient = [[OSSClient alloc] initWithEndpoint:OSS_ENDPOINT credentialProvider:pCredential];
    
    // generate the signed URL for the get request
    OSSTask *downloadURLTask = [_mClient presignConstrainURLWithBucketName:@"aliyun-dhc-shanghai" withObjectKey:OSS_DOWNLOAD_FILE_NAME withExpirationInterval:1800];
    [downloadURLTask waitUntilFinished];
    _downloadURLString = downloadURLTask.result;
    
    // generate the signed URL for the head request
    OSSTask *headURLTask = [_mClient presignConstrainURLWithBucketName:@"aliyun-dhc-shanghai" withObjectKey:OSS_DOWNLOAD_FILE_NAME httpMethod:@"HEAD" withExpirationInterval:1800 withParameters:nil];
    [headURLTask waitUntilFinished];
    
    _headURLString = headURLTask.result;
}

- (IBAction)resumeDownloadClicked:(id)sender {
    _downloadRequest = [DownloadRequest new];
    _downloadRequest.sourceURLString = _downloadURLString;       // set the URL of the resource
    _downloadRequest.headURLString = _headURLString;
    NSString *documentPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES).firstObject;
    _downloadRequest.downloadFilePath = [documentPath stringByAppendingPathComponent:OSS_DOWNLOAD_FILE_NAME];   //set the local storage path of the downloaded file
    
    __weak typeof(self) wSelf = self;
    _downloadRequest.downloadProgress = ^(int64_t bytesReceived, int64_t totalBytesReceived, int64_t totalBytesExpectToReceived) {
        // totalBytesReceived is the number of bytes that the client currently has cached, and totalBytesExpectToReceived is the total number of bytes that need to be downloaded.
        dispatch_async(dispatch_get_main_queue(), ^{
            __strong typeof(self) sSelf = wSelf;
            CGFloat fProgress = totalBytesReceived * 1.f / totalBytesExpectToReceived;
            sSelf.progressLab.text = [NSString stringWithFormat:@"%.2f%%", fProgress * 100];
            sSelf.progressBar.progress = fProgress;
        });
    };
    _downloadRequest.failure = ^(NSError *error) {
        __strong typeof(self) sSelf = wSelf;
        sSelf.checkpoint = error.userInfo[@"checkpoint"];
    };
    _downloadRequest.success = ^(NSDictionary *result) {
        NSLog(@"Download successful");
    };
    _downloadRequest.checkpoint = self.checkpoint;
    
    NSString *titleText = [[_downloadButton titleLabel] text];
    if ([titleText isEqualToString:@"download"]) {
        [_downloadButton setTitle:@"pause" forState: UIControlStateNormal];
        _downloadService = [DownloadService downloadServiceWithRequest:_downloadRequest];
        [_downloadService resume];
    } else {
        [_downloadButton setTitle:@"download" forState: UIControlStateNormal];
        [_downloadService pause];
    }
}

- (IBAction)cancelDownloadClicked:(id)sender {
    [_downloadButton setTitle:@"download" forState: UIControlStateNormal];
    [_downloadService cancel];
}

This section is the call in an upper layer service. The checkpoint can be obtained from the failure callback for both pausing and canceling the upload. The checkpoint can be passed to the DownloadRequest when resuming the download, which is used by the DownloadService to perform a consistency check.

For implementing the multipart download of OSS Object in Android, see the open source project The Multipart Download Function Implemented Based On Okhttp3. The following shows how to use this sample project to download OSS resources.

//1. First use SDK to get the download URL of the object
String signedURLString = ossClient.presignConstrainedObjectURL(bucket, object, expires);

//2. Add the download task

        mDownloadManager = DownloadManager.getInstance();
        mDownloadManager.add(signedURLString, new DownloadListner() {
            @Override
            public void onFinished() {
                Toast.makeText(MainActivity.this, "Download successful", Toast.LENGTH_SHORT).show();
            }

            @Override
            public void onProgress(float progress) {
                pb_progress1.setProgress((int) (progress * 100));
                tv_progress1.setText(String.format("%.2f", progress * 100) + "%");
            }

            @Override
            public void onPause() {
                Toast.makeText(MainActivity.this, "Paused!", Toast.LENGTH_SHORT).show();
            }

            @Override
            public void onCancel() {
                tv_progress1.setText("0%");
                pb_progress1.setProgress(0);
                btn_download1.setText("Download");
                Toast.makeText(MainActivity.this, "Download canceled!", Toast.LENGTH_SHORT).show();
            }
        });
        
//3. Start the download
        mDownloadManager.download(signedURLString);
        
//4. Pause the download
        mDownloadManager.cancel(signedURLString);
        
//5. Resume the download
        mDownloadManager.download(signedURLString);

Improvements

For If-Range in HTTP 1.1, let's look at the description of the header:

The If-Range HTTP request header makes a range request conditional: if the condition is fulfilled, the range request will be issued and the server sends back a 206 Partial Content answer with the appropriate body. If the condition is not fulfilled, the full resource is sent back, with a 200 OK status.

This header can be used either with a Last-Modified validator, or with an ETag, but not with both.

The most common use case is to resume a download, to guarantee that the stored resource has not been modified since the last fragment has been received.

Advantages of using If-Range: The client only needs one network request. When the condition is determined as failure, the If-Unmodified-Since or If-Match mentioned above will return 412 pre-condition check failure status code, and the client has to make another request to get the resource.

OSS Server currently does not support the field If-Range. When using NSURLSessionDownloadTask in iOS for multipart download, If-Range will be sent to the server to verify whether the file has been modified. So currently NSURLSessionDownloadTask cannot be used for multipart download.

References

https://www.alibabacloud.com/help/doc-detail/31856.htm

Community

Implementing OSS Multipart Download for Mobile Devices

Technical Points

Best Practices

Improvements

References

Read previous post:

Read next post:

Alibaba Cloud Storage

You may also like

Comments

Alibaba Cloud Storage

Related Products

Black Friday Cloud Services Sale

OSS(Object Storage Service)

Livestreaming for E-Commerce Solution