Object Storage Service:Configure client-side encryption by using OSS SDK for Python

Disclaimer

• When you use client-side encryption, you must ensure the integrity and validity of the CMK. If the CMK is incorrectly used or lost due to improper maintenance, you are responsible for all losses and consequences caused by decryption failures.

• When you copy or migrate encrypted data, you are responsible for the integrity and validity of object metadata. If the encrypted metadata is incorrect or lost due to improper maintenance, you are responsible for all losses and consequences caused by data decryption failures.

Scenarios

• Highly sensitive data: For personally identifiable information (PII), financial transaction records, health data, and other highly sensitive data, users may want to encrypt the data before the data leaves the local environment to make sure that the original data is effectively protected even if it is intercepted during transmission.

• Compliance requirements: Certain industries and regulations, such as the Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR), require strict encryption management for data stored on third-party platforms. Client-side encryption helps meet these compliance requirements because CMKs are managed by users and not transmitted over the network or exposed to cloud service providers.

• Full encryption control: An enterprise or developer may want to have full control over the encryption process, including selecting encryption algorithms and managing and rotating CMKs. In this case, you can use client-side encryption to ensure that only authorized users can decrypt and access data.

• Security of cross-region data migration: Client-side encryption helps maintain data in the encrypted state before and after cross-region data migration. This amplifies the security of data transmission over the Internet.

Usage notes

• In this topic, the public endpoint of the China (Hangzhou) region is used. If you want to access OSS from other Alibaba Cloud services in the same region as OSS, use an internal endpoint. For more information about OSS regions and endpoints, see Regions and endpoints.

• In this topic, access credentials are obtained from environment variables. For more information about how to configure access credentials, see Configure access credentials.

• In this topic, an OSSClient instance is created by using an OSS endpoint. If you want to create an OSSClient instance by using custom domain names or Security Token Service (STS), see Initialization.

Background information

In client-side encryption, a random data key is generated for each object to perform symmetric encryption on the object. The client uses a CMK to encrypt the random data key. The encrypted data key is uploaded as a part of the object metadata and stored in the OSS server. When an encrypted object is downloaded, the client uses the CMK to decrypt the random data key, and then uses the decrypted data key to decrypt the object. To ensure data security, the CMK is used only on the client and is not transmitted over the network or stored on the server.

Encryption methods

You can use two types of CMKs for client-side encryption:

• KMS-managed CMKs

  When you use a CMK managed in Key Management Service (KMS) for client-side encryption, you must provide OSS SDK for Python with the CMK ID.

• RSA-based CMKs managed by yourself

  When you use a CMK managed by yourself for client-side encryption, you must send the public key and the private key of your CMK to OSS SDK for Python as parameters.

You can use the preceding encryption methods to prevent data leaks and protect your data on the client. Even if your data is leaked, the data cannot be decrypted by others.

For more information about client-side encryption, see Client-side encryption in OSS User Guide.

Client-side encryption V2 (recommended)

• Object metadata related to client-side encryption

  Parameter	Description	Required
  x-oss-meta-client-side-encryption-key	The encrypted data key. The encrypted data key is a string encrypted by using an RSA-based CMK or a KMS-managed CMK and encoded in Base64.	Yes
  x-oss-meta-client-side-encryption-start	The initialization vector generated randomly for data encryption. The initialization vector is a string encrypted by using an RSA-based CMK or a KMS-managed CMK and encoded in Base64.	Yes
  x-oss-meta-client-side-encryption-cek-alg	The algorithm used to encrypt data.	Yes
  x-oss-meta-client-side-encryption-wrap-alg	The algorithm used to encrypt the data key.	Yes
  x-oss-meta-client-side-encryption-matdesc	The description of the content encryption key (CEK) in the JSON format.	No
  x-oss-meta-client-side-encryption-unencrypted-content-length	The length of data before encryption. If content-length is not specified, this parameter is not generated.	No
  x-oss-meta-client-side-encryption-unencrypted-content-md5	The MD5 hash of data before encryption. If content-md5 is not specified, this parameter is not generated.	No
  x-oss-meta-client-side-encryption-data-size	The total size of the object that you want to upload by using multipart upload.	Conditional (required for multipart upload)
  x-oss-meta-client-side-encryption-part-size	The part size specified in the multipart upload.	Conditional (required for multipart upload)

• Create a bucket for client-side encryption

  Before you encrypt and upload an object or download and decrypt an object on the client, you must initialize a bucket instance. You can call the operations on the bucket to upload or download objects. In client-side encryption, the CryptoBucket class inherits the operations from the Bucket class. You can configure parameters to initialize a CryptoBucket instance in the same manner you initialize a Bucket instance.

  • Initialize a CryptoBucket instance for object encryption by using an RSA-based CMK managed by yourself

    Important

    If you use an RSA-based CMK, you must manage the CMK by yourself. The loss of the CMK or the damage to the CMK data may cause decryption failures. We recommend that you use CMKs managed by KMS. If you need to use an RSA-based CMK to perform encryption, we recommend that you back up your CMK data.

    The following sample code provides an example on how to initialize a CryptoBucket instance for object encryption by using an RSA-based CMK managed by yourself:

    # -*- coding: utf-8 -*-
    import os
    import oss2
    from oss2.credentials import EnvironmentVariableCredentialsProvider
    from oss2.crypto import RsaProvider
    
    # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
    auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
    
    # If you want only to decrypt objects, specify only the private key. 
    # key_pair = {'private_key': 'yourPrivateKey'}
    # If you want only to encrypt objects, specify only the public key. 
    # key_pair = {'public_key': 'yourPublicKey'}
    # If you want to encrypt and decrypt objects, you must specify both the public key and the private key. 
    key_pair = {'private_key': 'yourPrivateKey', 'public_key': 'yourPublicKey'}
    bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName',
                               crypto_provider=RsaProvider(key_pair))
    

  • Initialize a CryptoBucket instance for object encryption by using a CMK managed by KMS

    The following sample code provides an example on how to initialize a CryptoBucket instance for object encryption by using a CMK managed by KMS:

    # -*- coding: utf-8 -*-
    import os
    
    import oss2
    from oss2.crypto import AliKMSProvider
    from oss2.credentials import EnvironmentVariableCredentialsProvider
    
    # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
    auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
    kms_provider = AliKMSProvider(auth, 'yourRegion', 'yourCMKID')
    bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName', crypto_provider=kms_provider)
    

  • Use and manage multiple CMKs

    You may use different CMKs to encrypt objects uploaded to a bucket or decrypt objects downloaded from the same bucket. You can specify different descriptions for the CMKs and add the CMKs and the descriptions to the encryption-related information about the bucket. When data is decrypted, OSS SDK for Python automatically matches a CMK based on the description. The following sample code provides an example on how to provide multiple CMKs:

    # -*- coding: utf-8 -*-
    import os
    import oss2
    from oss2.crypto import RsaProvider
    from oss2.credentials import EnvironmentVariableCredentialsProvider
    
    # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
    auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
    
    # Create an RSA-based key pair. 
    key_pair_1 = {'private_key': 'yourPrivateKey_1', 'public_key': 'yourPublicKey_1'}
    mat_desc_1 = {'key1': 'value1'}
    
    # Create another RSA-based key pair. 
    key_pair_2 = {'private_key': 'yourPrivateKey_2', 'public_key': 'yourPublicKey_2'}
    mat_desc_2 = {'key2': 'value2'}
    
    provider = RsaProvider(key_pair=key_pair_2, mat_desc=mat_desc_2)
    # Add the description of key_pair_1 to the provider. 
    encryption_materials = oss2.EncryptionMaterials(mat_desc_1, key_pair=key_pair_1)
    provider.add_encryption_materials(encryption_materials)
    
    # Use the provider to initialize a CryptoBucket instance. Then, you can download the object data that is encrypted by using a CMK whose description is mat_desc_1. 
    crypto_bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName', crypto_provider=provider)
    

• Perform client-side encryption in simple upload and download

  The following sample code provides an example on how to use a CMK managed by KMS to encrypt an object when you upload the object in simple upload and decrypt the object when you download the object:

  # -*- coding: utf-8 -*-
  import os
  import oss2
  from oss2.credentials import EnvironmentVariableCredentialsProvider
  from oss2.crypto import RsaProvider
  from oss2.cryptoimportAliKMSProvider
  
  # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
  auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
  
  kms_provider = AliKMSProvider(auth, 'yourRegion', 'yourCMKID')
  bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName', crypto_provider=kms_provider)
  
  key = 'motto.txt'
  content = b'a' * 1024 * 1024
  filename = 'download.txt'
  
  
  # Upload the object. 
  bucket.put_object(key, content, headers={'content-length': str(1024 * 1024)})
  
  # Download the object from OSS to the local memory. 
  result = bucket.get_object(key)
  
  # Check data consistency between the content of the downloaded object and that of the object before upload. 
  content_got = b''
  for chunk in result:
      content_got += chunk
  assert content_got == content
  
  # Download the object from OSS to a local path. 
  result = bucket.get_object_to_file(key, filename)
  
  # Check data consistency between the content of the downloaded object and that of the object before upload. 
  with open(filename, 'rb') as fileobj:
      assert fileobj.read() == content

• Perform client-side encryption in multipart upload

  Note

  • If a multipart upload task is interrupted and the process is terminated, the context of the multipart upload task may be lost. If a multipart upload task is interrupted and you want to re-upload the object, you must re-upload the entire object.

  • We recommend that you call the operation for resumable upload in OSS to upload large objects. This way, you can store the context of a multipart upload task on the local client so that the context can be retained even if the upload task is interrupted.

  The following sample code provides an example on how to use a KMS-managed CMK to encrypt an object that you want to upload by using multipart upload:

  # -*- coding: utf-8 -*-
  import os
  import oss2
  from oss2.credentials import EnvironmentVariableCredentialsProvider
  from  oss2.crypto import RsaProvider
  from oss2.cryptoimportAliKMSProvider
  
  # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
  auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
  
  kms_provider=AliKMSProvider(auth, 'yourRegion', 'yourCMKID')
  bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName', crypto_provider = kms_provider)
  
  """
  Upload an object in parts.
  """
  # Initiate a multipart upload task. 
  part_a = b'a' * 1024 * 100
  part_b = b'b' * 1024 * 100
  part_c = b'c' * 1024 * 100
  multi_content = [part_a, part_b, part_c]
  
  parts = []
  data_size = 100 * 1024 * 3
  part_size = 100 * 1024
  multi_key = "test_crypto_multipart"
  
  # Initialize the context of the multipart upload task when you use client-side encryption. 
  context = models.MultipartUploadCryptoContext(data_size, part_size)
  res = bucket.init_multipart_upload(multi_key, upload_context=context)
  upload_id = res.upload_id
  
  # In this example, the parts are uploaded in sequence. Multipart upload supports multiple parallel threads to accelerate the upload. 
  for i in range(3):
      # The context values cannot be modified. If the values in context are modified, data upload fails. 
      result = bucket.upload_part(multi_key, upload_id, i + 1, multi_content[i], upload_context=context)
      parts.append(oss2.models.PartInfo(i + 1, result.etag, size=part_size, part_crc=result.crc))
  
  # Complete the multipart upload task. 
  result = bucket.complete_multipart_upload(multi_key, upload_id, parts)
  
  # Check data consistency between the content of the downloaded object and that of the object before upload. 
  result = bucket.get_object(multi_key)
  content_got = b''
  for chunk in result:
      content_got += chunk
  assert content_got[0:102400] == part_a
  assert content_got[102400:204800] == part_b
  assert content_got[204800:307200] == part_c

• Perform client-side encryption in resumable upload

  The following sample code provides an example on how to use an RSA-based CMK managed by yourself to encrypt an object that you want to upload by using resumable upload:

  # -*- coding: utf-8 -*-
  import os
  import oss2
  from oss2.credentials import EnvironmentVariableCredentialsProvider
  from  oss2.crypto import RsaProvider
  
  key = 'motto.txt'
  content = b'a' * 1024 * 1024 * 100
  file_name_put = 'upload.txt'
  
  # Write the data included in content to the file. 
  with open(file_name_put, 'wb') as fileobj:
      fileobj.write(content)
  
  # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
  auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
  
  # Initialize a CryptoBucket instance and use the RSA-based CMK to perform client-side encryption. This encryption method supports only uploads and downloads of entire objects. 
  # If you use OSS SDK for Python 2.9.0 or later, we recommend that you do not use LocalRsaProvider to initialize the CryptoBucket instance. 
  # bucket = oss2.CryptoBucket(auth,'yourEndpoint', 'yourBucketName', crypto_provider=LocalRsaProvider())
  
  # If you want only to decrypt objects, specify only the private key. 
  # key_pair = {'private_key': 'yourPrivateKey'}
  # If you want only to encrypt objects, specify only the public key. 
  # key_pair = {'public_key': 'yourPublicKey'}
  # If you want to encrypt and decrypt objects, you must specify both the public key and the private key. 
  key_pair = {'private_key': 'yourPrivateKey', 'public_key': 'yourPublicKey'}
  
  # Initialize a CryptoBucket instance. If you set upload_contexts_flag to True, you do not need to configure parameters for multipart upload context when you call the upload_part operation. 
  bucket = oss2.CryptoBucket(auth, endpoint, bucket_name,
                             crypto_provider=RsaProvider(key_pair))
  
  # In this example, the value of multipart_threshold is set to 10 * 1024 * 1024. The default multipart upload threshold is 10 MB. Specify a value based on your scenario. 
  # multipart_threshold specifies the object size threshold for using multipart upload. If the object size exceeds the threshold, multipart upload is used. If the object size is smaller than the threshold, we recommend that you call put_object to upload the object. 
  # part_size specifies the part size in multipart upload. The default part size is 10 MB. 
  # num_threads specifies the number of parallel upload threads. Default value: 1. 
  oss2.resumable_upload(bucket, key, file_name_put, multipart_threshold=10 * 1024 * 1024, part_size=1024 * 1024, num_threads=3)

• Perform client-side encryption in resumable download

  The following sample code provides an example on how to use a CMK managed by KMS to decrypt an object that you want to download in resumable download:

  # -*- coding: utf-8 -*-
  import os
  import oss2
  from oss2.crypto import RsaProvider
  from oss2.cryptoimportAliKMSProvider
  from oss2.credentials import EnvironmentVariableCredentialsProvider
  
  # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
  auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
  key = 'motto.txt'
  content = b'a' * 1024 * 1024 * 100
  file_name_get = 'download.txt'
  
  kms_provider = AliKMSProvider(auth, 'yourRegion', 'yourCMKID')
  bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName', crypto_provider=kms_provider)
  
  
  # Download an object by using resumable download. 
  oss2.resumable_download(bucket, key, file_name_get, multiget_threshold=10 * 1024 * 1024, part_size=1024 * 1024, num_threads=3)
  
  # Check data consistency between the content of the downloaded object and that of the object before upload. 
  with open(file_name_get, 'rb') as fileobj:
      assert fileobj.read() == content
  

Client-side encryption V1 (not recommended)

• Object metadata related to client-side encryption

  Parameter	Description	Required
  x-oss-meta-oss-crypto-key	The encrypted data key. The encrypted data key is a string encrypted by using an RSA-based CMK and encoded in Base64.	Yes
  x-oss-meta-oss-crypto-start	The initial value generated randomly for data encryption. The value is a string encrypted by using an RSA-based CMK and encoded in Base64.	Yes
  x-oss-meta-oss-cek-alg	The algorithm used to encrypt data.	Yes
  x-oss-meta-oss-wrap-alg	The algorithm used to encrypt the data key.	Yes
  x-oss-meta-oss-matdesc	The description of the content encryption key (CEK) in the JSON format. This parameter does not take effect.	No
  x-oss-meta-unencrypted-content-length	The length of data before encryption. If content-length is not specified, this parameter is not generated.	No
  x-oss-meta-unencrypted-content-md5	The MD5 hash of data before encryption. If content-md5 is not specified, this parameter is not generated.	No

• Use RSA-based CMKs managed by yourself to perform client-side encryption for object uploads and downloads

  The following sample code provides an example on how to use an RSA-based CMK to encrypt an object to upload and decrypt the object to download:

  # -*- coding: utf-8 -*-
  import os
  import oss2
  from oss2.credentials import EnvironmentVariableCredentialsProvider
  from oss2.crypto import LocalRsaProvider
  
  # Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
  auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
  
  # Initialize a CryptoBucket instance and use the RSA-based CMK to perform client-side encryption. This encryption method supports only uploads and downloads of entire objects. 
  bucket = oss2.CryptoBucket(auth, 'yourEndpoint', 'yourBucketName', crypto_provider=LocalRsaProvider())
  
  key = 'motto.txt'
  content = b'a' * 1024 * 1024
  filename = 'download.txt'
  
  
  # Upload the object. 
  bucket.put_object(key, content, headers={'content-length': str(1024 * 1024)})
  
  # Download the object from OSS to the local memory. 
  result = bucket.get_object(key)
  
  # Check data consistency between the content of the downloaded object and that of the object before upload. 
  content_got = b''
  for chunk in result:
      content_got += chunk
  assert content_got == content
  
  # Download the object from OSS to a local path. 
  result = bucket.get_object_to_file(key, filename)
  
  # Check data consistency between the content of the downloaded object and that of the object before upload. 
  with open(filename, 'rb') as fileobj:
      assert fileobj.read() == content

References

For the complete sample code for client-side encryption, visit GitHub.