All Products
Search
Document Center

Object Storage Service:CRR within the same account

Last Updated:Oct 14, 2024

Cross-region replication (CRR) within the same account allows the automatic and asynchronous (near real-time) replication of Object Storage Service (OSS) objects from a bucket in a region to another bucket in another region within the same Alibaba Cloud account. CRR within the same account replicates operations, such as the creation, overwriting, and deletion of objects, from a source bucket to a destination bucket. This topic describes how to perform CRR within the same account.

Prerequisites

  • Bucket A, which works as the source bucket of a CRR task, is created in a region within an Alibaba Cloud account. The ID of the Alibaba Cloud account, the name of Bucket A, and the region in which Bucket A is located are recorded.

  • Bucket B, which works as the destination bucket of the CRR task, is created in another region within the same Alibaba Cloud account. The name of Bucket B and the region in which Bucket B is located are recorded.

Role types

When you perform CRR within the same account, you must specify the role that is used to replicate objects between the source and destination buckets. You can select one of the following RAM roles to perform CRR within the same account.

Important

You can create a RAM role by using a RAM user. The RAM user must have the following permissions: ram:CreateRole, ram:GetRole, ram:ListPoliciesForRole, and ram:AttachPolicyToRole. However, it is risky to grant a RAM user permissions such as ram:CreateRole and ram:GetRole. You can use the Alibaba Cloud account with which the RAM user is associated to create a RAM role and grant the required permissions to the RAM role. Then, the RAM user can assume the RAM role that is created by the Alibaba Cloud account.

(Recommended) New RAM role

When you create a rule to perform CRR within the same account, you can create a RAM role. After you select New RAM Role from the RAM Role Name drop-down list, a RAM role in the oss-replication-{uuid} format is automatically created and different policies are attached to the role based on whether you set Replicate Objects Encrypted based on KMS to Yes.

  • Set Replicate Objects Encrypted based on KMS to Yes

    After you create the RAM role, follow the on-screen instructions to attach the required policies to the role. Then, the RAM role is attached to the policy to replicate data from the source bucket to the destination bucket and the AliyunKMSFullAccess policy to manage Key Management Service (KMS).

  • Set Replicate Objects Encrypted based on KMS to No

    After you create the RAM role, follow the on-screen instructions to attach the required policies to the role. Then, the RAM role is attached to the policy to replicate data from the source bucket to the destination bucket.

AliyunOSSRole

When you create a rule to perform CRR within the same account, you can select the AliyunOSSRole role to complete the data replication task. After you select AliyunOSSRole, different policies are attached to the AliyunOSSRole role based on whether you set Replicate Objects Encrypted based on KMS to Yes.

  • Set Replicate Objects Encrypted based on KMS to Yes

    After you select AliyunOSSRole, the following policies are automatically attached to the AliyunOSSRole role: AliyunOSSFullAccess (permissions to manage OSS) and AliyunKMSFullAccess (permissions to manage KMS).

    Warning

    The AliyunOSSRole role has the permissions to perform all operations on all buckets within the current Alibaba Cloud account and KMS. Exercise caution when you use this role.

  • Set Replicate Objects Encrypted based on KMS to No

    After you select AliyunOSSRole, the AliyunOSSFullAccess policy is automatically attached to the AliyunOSSRole role.

    Warning

    The AliyunOSSRole role has the permissions to perform all operations on all buckets within the current Alibaba Cloud account. Exercise caution when you use this role.

Custom role

When you create a rule to perform CRR within the same account, you can use a custom role to complete the data replication task. You need to create a custom role in the RAM console and attach the required policies to the custom role.

  1. Create a normal service role.

    When you create the role, set Select Trusted Entity to Alibaba Cloud Service, Role Type to Normal Service Role, and then Select Trusted Service to OSS. For more information, see Create a regular service role.

  2. Grant permissions to the RAM role.

    You can use one of the following methods to grant the permissions to the RAM role:

    Attach the required system policy to the RAM role

    Warning

    Attach the AliyunOSSFullAccess system policy to the RAM role. The RAM role to which the AliyunOSSFullAccess system policy is attached has the permissions to perform all operations on all buckets within the current Alibaba Cloud account. Proceed with caution when you attach the system policy to the RAM role.

    If you want to replicate objects encrypted by KMS to the destination bucket, you must attach the AliyunKMSFullAccess system policy to the RAM role.

    For more information, see Grant permissions to a RAM role.

    Attach a custom policy to the RAM role

    You can attach a custom policy to the RAM role to grant the role the minimum permissions required to perform CRR on the source bucket and the destination bucket.

    Note

    Replace src-bucket and dest-bucket with the name of the source bucket and the name of the destination bucket based on your business requirements.

    {
       "Version":"1",
       "Statement":[
          {
             "Effect":"Allow",
             "Action":[
                "oss:ReplicateList",
              	"oss:ReplicateGet"
             ],
             "Resource":[
              	"acs:oss:*:*:src-bucket",
                "acs:oss:*:*:src-bucket/*"
             ]
          },
          {
             "Effect":"Allow",
             "Action":[
              	"oss:ReplicateList",
                "oss:ReplicateGet",
                "oss:ReplicatePut",
                "oss:ReplicateDelete"
             ],
             "Resource":[
              	"acs:oss:*:*:dest-bucket",
                "acs:oss:*:*:dest-bucket/*"
             ]
          }
       ]
    }

    For more information, see Grant permissions to a RAM role.

    Note

    If you want to replicate objects encrypted by KMS to the destination bucket, you must attach the AliyunKMSFullAccess system policy to the RAM role.

Methods

Use the OSS console

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, click the name of the source bucket.

  3. In the left-side navigation tree, choose Data Management > CRR.

  4. On the CRR tab, click CRR.

  5. In the CRR panel, configure the parameters. The following table describes the parameters.

    Section

    Parameter

    Description

    Configure Destination Bucket

    Source Bucket

    The region and name of the source bucket are displayed. You do not need to specify the parameter.

    Destination Bucket

    Select Select a bucket that belongs to the Alibaba Cloud account, select a region from the Region drop-down list, and then select the destination bucket from the Bucket drop-down list.

    Configure Replication Policy

    Objects to Replicate

    Select the objects that you want to replicate to the destination bucket. Valid values:

    Note

    After you create a data replication rule, changes to the x-oss-last-access-time attribute of the objects and the storage class conversion of objects in the source bucket due to lifecycle rules or the CopyObject operation are not synchronized to the destination bucket.

    • All Objects in Source Bucket: OSS replicates all objects from the source bucket to the destination bucket.

    • Objects with Specified Prefix: OSS replicates the objects whose names contain a specific prefix from the source bucket to the destination bucket. You can specify up to 10 prefixes.

    Replication Policy

    Configure the data replication mode. Valid values:

    • Add/Change: OSS replicates only object creation and update operations from the source bucket to the destination bucket.

    • Add/Delete/Change: OSS replicates object creation, update, and deletion operations from the source bucket to the destination bucket.

    If you perform multipart upload to upload an object to the source bucket, each uploaded part is replicated to the destination bucket. The complete object that is obtained by calling the CompleteMultipartUpload operation is also replicated to the destination bucket.

    For more information, see Data replication in specific scenarios.

    Replicate Historical Data

    Specify whether to replicate historical data (data that exists in the source bucket before you enable CRR) to the destination bucket.

    • Yes: Historical data is replicated to the destination bucket.

      Important

      When historical data is replicated, objects that are replicated from the source bucket may overwrite objects that have the same names in the destination bucket. To prevent data loss, we recommend that you enable versioning for the source and destination buckets.

    • No: OSS replicates only objects that are uploaded or updated after the CRR rule takes effect to the destination bucket.

    Replicate Objects Encrypted Based on KMS

    Specify whether to replicate objects encrypted based on KMS to the destination bucket.

    • Yes: If KMS-based encryption is configured for objects in the source or destination bucket, the objects are replicated to the destination bucket.

      Note

      You can call the HeadObject operation to query the encryption rules of objects in the source bucket and the GetBucketEncryption operation to query the encryption rules of the destination bucket.

    • No: Objects that are encrypted based on KMS are not replicated to the destination bucket.

    CMK ID

    Specify the customer master key (CMK) that is used to encrypt destination objects.

    If you want to use a CMK to encrypt objects, you must create a CMK in the same region as the destination bucket in the KMS console. For more information, see Create a CMK.

    RAM Role Name

    Specify the RAM role that you want to use to replicate objects. Select New RAM Role. This is the recommended option. After you select New RAM Role from the drop-down list, follow the on-screen instructions to grant permissions to the new RAM role.

    You can also select AliyunOSSRole or a custom RAM role. For more information about the three types of RAM roles, see Role types.

    Configure Replication Speed

    Acceleration Type

    Specify the acceleration type. Only Transfer Acceleration is supported. You can use transfer acceleration to accelerate data transfer when you replicate data across regions in the Chinese mainland and outside the Chinese mainland. If you enable transfer acceleration, you are charged transfer acceleration fees. For more information, see Transfer acceleration fees.

    RTC

    Note

    RTC is available in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), and China (Shenzhen).

    RTC is available in the following regions: US (Silicon Valley) and US (Virginia).

    After Replication Time Control (RTC) is enabled, OSS replicates most of the objects that you uploaded to OSS within a few seconds and replicates 99.99% of objects within 10 minutes. You are charged when you enable RTC. For more information, see RTC traffic fees.

    For more information about the regions in which RTC is supported, see RTC.

  6. Click OK. In the message that appears, click Enable.

    • After you configure a CRR rule, you cannot modify or delete the rule.

    • After you configure a CRR rule, the replication task starts in 3 to 5 minutes. You can view the replication progress on the CRR tab of the source bucket.

    • In CRR, data is asynchronously replicated. The period of time that is required to replicate data from the source bucket to the destination bucket varies based on the amount of data. The period of time may range from a few minutes to a few hours.

Use OSS SDKs

CRR within the same account is supported only by using OSS SDK for Java, OSS SDK for Python, and OSS SDK for Go.

Java

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.AddBucketReplicationRequest;

public class Demo {

    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // We recommend that you do not save access credentials in the project code. Otherwise, access credentials may be leaked. As a result, the security of all resources in your account is compromised. In this example, access credentials are obtained from environment variables. Before you run the sample code, make sure that the environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the source bucket. 
        String bucketName = "src-bucket";
        // Specify the name of the destination bucket. The source and destination buckets must belong to the same Alibaba Cloud account. 
        String targetBucketName = "dest-bucket";
        // Specify the region in which the destination bucket is located. The source and destination buckets must be located in different regions. 
        String targetBucketLocation = "oss-cn-shanghai";

        // Create an OSSClient instance. 
        OSS ossClient = new OSSClientBuilder().build(endpoint, credentialsProvider);

        try {
            AddBucketReplicationRequest request = new AddBucketReplicationRequest(bucketName);
            request.setTargetBucketName(targetBucketName);
            request.setTargetBucketLocation(targetBucketLocation);
            // Specify whether to replicate historical data. By default, historical data is replicated. In this example, this parameter is set to false, which indicates that historical data is not replicated. 
            request.setEnableHistoricalObjectReplication(false);
            // Specify the name of the RAM role that you want OSS to use to replicate data. The role must have the permissions to perform CRR on the source bucket and receive replicated objects in the destination bucket. 
            request.setSyncRole("yourRole");
            // Specify whether to replicate the objects that are encrypted by using SSE-KMS. 
            //request.setSseKmsEncryptedObjectsStatus("Enabled");
            // Specify the CMK ID used in SSE-KMS. If Status is set to Enabled, you must specify this parameter. 
            //request.setReplicaKmsKeyID("3542abdd-5821-4fb5-a425-90adca***");
            //List prefixes = new ArrayList();
            //prefixes.add("image/");
            //prefixes.add("video");
            //prefixes.add("a");
            //prefixes.add("A");
            // Specify the prefixes that are contained in the names of the objects that you want to replicate. After you specify the prefixes, only objects whose names contain one of the prefixes are replicated to the destination bucket. 
            //request.setObjectPrefixList(prefixes);
            //List actions = new ArrayList();
            //actions.add(AddBucketReplicationRequest.ReplicationAction.ALL);
            // Specify the operations that you want to replicate to the destination bucket. The default value is ALL, which indicates that all operations performed on objects in the source bucket are replicated to the destination bucket. 
            //request.setReplicationActionList(actions);
            ossClient.addBucketReplication(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}        

Python

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider
from oss2.models import ReplicationRule
# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. 
# Specify the name of the source bucket. Example: src-bucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'src-bucket')
replica_config = ReplicationRule(
    # Specify the name of the destination bucket. The source and destination buckets must belong to the same Alibaba Cloud account. 
    target_bucket_name='dest-bucket',
    # Specify the region in which the destination bucket is located. The source and destination buckets must be located in different regions. 
    target_bucket_location='oss-cn-shanghai',
    # Specify the name of the RAM role that you want OSS to use to replicate data. The role must have the permissions to perform CRR on the source bucket and receive replicated objects in the destination bucket. 
    sync_role_name='roleNameTest',
)

# Specify the prefixes that are contained in the names of the objects that you want to replicate. After you specify the prefixes, only objects whose names contain one of the prefixes are replicated to the destination bucket. 
# prefix_list = ['prefix1', 'prefix2']
# Specify the data replication rule. 
# replica_config = ReplicationRule(
     # prefix_list=prefix_list,
     # Specify the operations that you want to replicate to the destination bucket. The default value is ALL, which indicates that all operations performed on objects in the source bucket are synchronized to the destination bucket. 
     # action_list=[ReplicationRule.ALL],
     # Specify the name of the destination bucket. The source and destination buckets must belong to the same Alibaba Cloud account. 
     # target_bucket_name='dest-bucket',
     # Specify the region in which the destination bucket is located. The source and destination buckets must be located in different regions. 
     # target_bucket_location='yourTargetBucketLocation',
     # Specify whether to replicate historical data. By default, historical data is replicated. In this example, this parameter is set to False, which indicates that historical data is not replicated. 
     # is_enable_historical_object_replication=False,
     # Specify the link that you want to use to transfer data during data replication. 
     # target_transfer_type='oss_acc',    
  #)

# Enable data replication. 
bucket.put_bucket_replication(replica_config)

Go

package main

import (
	"encoding/xml"
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func HandleError(err error) {
	fmt.Println("Error:", err)
	os.Exit(-1)
}

// Enable data replication. 
func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
	// Specify the name of the source bucket. 
	srcbucketName := "yourSrcBucket"
	// Specify the name of the destination bucket. 
	destBucketName := "yourDestBucket"
	// Specify that objects whose names contain one of the following prefixes are replicated to the destination bucket: prefix_1 and prefix_2.  
	// To replicate all objects from the source bucket to the destination bucket, do not configure prefixes. 
	prefix1 := "prefix_1"
	prefix2 := "prefix_2"
	// Specify the CMK ID used in SSE-KMS. If Status is set to Enabled, you must specify this parameter. 
	keyId := "c4d49f85-ee30-426b-a5ed-95e9xxxx"
	// Specify whether to replicate the objects that are encrypted by using SSE-KMS. 
	source := "Enabled"
	prefixSet := oss.ReplicationRulePrefix{Prefix: []*string{&prefix1, &prefix2}}
	// Enable the RTC feature. 
	enabled := "enabled"
	reqReplication := oss.PutBucketReplication{
		Rule: []oss.ReplicationRule{
			{
				PrefixSet: &prefixSet,
				// Specify the operations that you want to replicate to the destination bucket. The default value is ALL, which indicates that all operations performed on objects in the source bucket are synchronized to the destination bucket. 
				Action: "ALL",
				RTC:    &enabled,
				Destination: &oss.ReplicationRuleDestination{
					Bucket: destBucketName,
					// Specify the region in which the destination bucket is located. The source bucket and destination bucket must be located in different regions. 
					Location: "oss-cn-shanghai",
					// Specify the link that you want to use to transfer data during data replication. In this example, this parameter is set to oss_acc, which indicates that the link used for transfer acceleration is used. 
					TransferType: "oss_acc",
				},
				// Specify whether to replicate historical data. By default, historical data is replicated. In this example, this parameter is set to disabled, which indicates that historical data is not replicated. 
				HistoricalObjectReplication: "disabled",
				// Specify the name of the RAM role that you want OSS to use to replicate data. The role must have the permissions to perform CRR on the source bucket and receive replicated objects in the destination bucket. 
				SyncRole:                "yourRole",
				EncryptionConfiguration: &keyId,
				SourceSelectionCriteria: &source,
			},
		},
	}

	xmlBody, err := xml.Marshal(reqReplication)
	if err != nil {
		HandleError(err)
	}
	err = client.PutBucketReplication(srcbucketName, string(xmlBody))

	if err != nil {
		HandleError(err)
	}

	fmt.Println("Put Bucket Replication Success!")
}

Use ossutil

For more information about how to use ossutil to perform CRR within the same account, see replication.

Use the OSS API

If your business requires a high level of customization, you can directly call the RESTful APIs. To directly call an API, you must include the signature calculation in your code. For more information, see PutBucketReplication.

References