Object Storage Service:Check data integrity by using CRC-64

Usage notes

• The PutObject, AppendObject, PostObject, and MultipartUploadPart operations return a corresponding CRC-64 value. The client can obtain the CRC-64 value returned by the server after the upload is complete and can check it against the value calculated on the local machine.

• When the MultipartComplete operation is called, the CRC-64 value of the entire object is returned if each part has a CRC-64 value. If one of the parts does not have a CRC-64 value, OSS does not return the CRC-64 value of the final object. For example, if a part is uploaded before the CRC-64 checksum is supported, no CRC-64 value is returned for the part and for the final object.

• The GetObject, HeadObject, and GetObjectMeta operations return the corresponding CRC-64 value (if any). After the GetObject operation is complete, the client can obtain the CRC-64 value returned by the server and check it against the value calculated on the local machine.

  Note

  The GET request that includes the Range header returns the CRC-64 value of the entire object.

• The destination objects generated from copy operations such as CopyObject and UploadPartCopy may not have a CRC-64 value.

Examples

The following sample code in Python provides an example on how to use CRC-64 values to verify data integrity.

import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
import crcmod
import random
import string
from oss2.models import PartInfo

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
# Specify the name of the bucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'yourBucketName')
# Create a CRC-64 verification function. 
do_crc64 = crcmod.mkCrcFun(0x142F0E1EBA9EA3693, initCrc=0, xorOut=0xffffffffffffffff, rev=True)

# Verify the CRC-64 values and display the result. 
def check_crc64(local_crc64, oss_crc64, msg="check crc64"):
    if local_crc64 != oss_crc64:
        print("{0} check crc64 failed. local:{1}, oss:{2}.".format(msg, local_crc64, oss_crc64))
        return False
    else:
        print("{0} check crc64 ok.".format(msg))
        return True

# Generate a random string of the specified length. 
def random_string(length):
    return ''.join(random.choice(string.ascii_lowercase) for i in range(length))

# Generate a random string with a length of 1024. 
content = random_string(1024)

# Specify the path of the object. 
key = 'normal-key'

# Verify data integrity for a GetObject operation. 
result = bucket.put_object(key, content)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(content)))
check_crc64(local_crc64, oss_crc64, "put object")

# Verify data integrity for a GetObject operation. 
result = bucket.get_object(key)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(result.resp.read()))
check_crc64(local_crc64, oss_crc64, "get object")

# Verify data integrity for parts and the final object. 
part_info_list = []
key = "multipart-key"
result = bucket.init_multipart_upload(key)
upload_id = result.upload_id
part_1 = random_string(1024 * 1024)
result = bucket.upload_part(key, upload_id, 1, part_1)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_1)))
# Check the integrity of part 1. 
check_crc64(local_crc64, oss_crc64, "upload_part object 1")
part_info_list.append(PartInfo(1, result.etag, len(part_1)))
part_2 = random_string(1024 * 1024)
result = bucket.upload_part(key, upload_id, 2, part_2)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_2)))
# Check the integrity of part 2. 
check_crc64(local_crc64, oss_crc64, "upload_part object 2")
part_info_list.append(PartInfo(2, result.etag, len(part_2)))
result = bucket.complete_multipart_upload(key, upload_id, part_info_list)
oss_crc64 = result.headers.get('x-oss-hash-crc64ecma', '')
local_crc64 = str(do_crc64(oss2.to_bytes(part_2), do_crc64(oss2.to_bytes(part_1))))
# Check whether the final object uploaded to OSS is identical to the local file. 
check_crc64(local_crc64, oss_crc64, "complete object")

Supported OSS SDKs

The following table describes OSS SDKs that support CRC-64 for downloads and uploads.