All Products
Search
Document Center

Object Storage Service:Mirroring-based back-to-origin

Last Updated:Sep 29, 2024

After you migrate all existing data from your server to an Object Storage Service (OSS) bucket, you can configure a mirroring-based back-to-origin rule to allow requesters to access data that is available on your server but has not been migrated to the bucket. When a requester requests a resource that does not exist in the bucket, OSS attempts to retrieve the resource from the origin that is specified in the mirroring-based back-to-origin rule. If the requested resource is retrieved from the origin, OSS stores the resource in the bucket and returns the resource to the requester.

Limits

  • Regions

    You can configure mirroring-based back-to-origin rules for buckets that are located in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Ulanqab), China (Shenzhen), China (Heyuan), China (Guangzhou), China (Chengdu), China (Hong Kong), US (Silicon Valley), US (Virginia), Japan (Tokyo), Singapore, Australia (Sydney) Closing Down, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Philippines (Manila), Germany (Frankfurt), UK (London), and UAE (Dubai).

  • Number of mirroring-based back-to-origin rules

    You can configure up to 20 mirroring-based back-to-origin rules for a bucket. The rules are sequentially matched with requests based on the values of RuleNumber. If a mirroring-based back-to-origin rule is matched, the subsequent rules are ignored. OSS determines whether a request matches a mirroring-based back-to-origin rule based on whether the request meets the conditions specified in the rule. OSS does not check whether the requested object can be obtained from the origin.

  • QPS and traffic

    In regions in the Chinese mainland, the default maximum queries per second (QPS) for mirroring-based back-to-origin requests and default maximum bandwidth for an Alibaba Cloud account are 2,000 and 2 Gbit/s. In regions outside the Chinese mainland, the default maximum QPS for mirroring-based back-to-origin requests and default maximum bandwidth for an Alibaba Cloud account are 1,000 and 1 Gbit/s.

    If your business requires higher QPS or bandwidth, contact technical support.

  • Origin URLs

    Origin URLs cannot be URLs on an internal network.

  • Default timeout period

    By default, the timeout period of a mirroring-based back-to-origin request is 10 seconds.

Scenarios

Mirroring-based back-to-origin is used to seamlessly migrate data to OSS. For example, you can use mirroring-based back-to-origin to migrate business from a self-managed origin or another cloud service to OSS without business interruption. During the migration, you can use mirroring-based back-to-origin rules to obtain the data that has not been migrated to OSS. This ensures business continuity. For more information about use cases, see Seamlessly migrate data of a web-based service provider to OSS.

Note

Mirroring-based back-to-origin rules do not generate additional fees. However, a mirroring-based back-to-origin request generates API operation calling fees. You are charged for a mirroring-based back-to-origin request in the same manner that you are charged for other requests. For more information, see API operation calling fees.

Process

The following figure shows how mirroring-based back-to-origin works.

image

Back-to-origin rules

  • Trigger conditions

    OSS obtains an object from the origin based on the mirroring-based back-to-origin rule only if HTTP status code 404 is returned for the GetObject operation.

  • Naming rules for requested objects

    The URL that OSS uses to obtain an object from the origin is in the following format: http(s)://MirrorURL/ObjectName. ObjectName specifies the name of the requested object. For example, if the back-to-origin URL configured for a bucket is https://aliyun.com and the requested object named example.jpg does not exist in the bucket, OSS obtains the object from https://aliyun.com/example.jpg and stores the object as example.jpg in OSS.

  • Failed back-to-origin requests

    If the requested object does not exist in the origin, the origin returns HTTP status code 404 to OSS. Then, OSS returns the same HTTP status code to the requester. If the origin returns an HTTP status code other than 200 and 404, the back-to-origin request fails due to factors such as network errors. In this case, OSS returns the 424 MirrorFailed error code to the requester.

  • Updates of objects obtained from an origin

    After OSS obtains an object by using mirroring-based back-to-origin, if the object at the origin is modified, OSS does not update the obtained object.

  • Metadata of objects obtained from an origin

    OSS stores the following HTTP headers returned by an origin as the metadata of OSS objects:

    Content-Type
    Content-Encoding
    Content-Disposition
    Cache-Control
    Expires
    Content-Language
    Access-Control-Allow-Origin
  • HTTP requests

    • QueryString and headers that are contained in the request sent to OSS are not contained in the request sent by OSS to an origin. OSS determines whether to send the information to the origin based on the configured back-to-origin rules.

    • If the origin returns chunked-encoded data, OSS returns chunked-encoded data.

Methods

Use the OSS console

If you configure multiple back-to-origin rules for a bucket in the OSS console, the rules take effect in chronological order based on the time when the rules were created. You can click Up or Down on the right side of the rules to change the priority of the rules.

3

When a requester requests an object that does not exist in the bucket, OSS attempts to retrieve the object from the origin based on the configured origin URL and back-to-origin conditions. For example, the examplefolder subdirectory exists in the root directory of the examplebucket bucket in the China (Hangzhou) region. If a requested object does not exist in the examplefolder subdirectory, you want OSS to search the examplefolder directory at https://www.example.com/ for the requested object and return the object to the requester. To achieve this goal, perform the following steps to configure a back-to-origin rule:

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

  3. In the left-side navigation tree, choose Data Management > Mirroring-based Back-to-origin.

  4. On the Mirroring-based Back-to-origin page, click Create Rule.

  5. In the Create Rule panel, configure the required parameters described in the following table and retain the default settings for other parameters.

    Parameter

    Description

    Method

    Select Mirroring.

    Condition

    Select Object Name Prefix and set the parameter to examplefolder/.

    Note

    If you configure only one mirroring-based back-to-origin rule for the bucket, you can leave the Object Name Prefix parameter and the File Name Suffix parameter empty. If you configure multiple mirroring-based back-to-origin rules for the bucket, you must specify different name prefixes or suffixes to differentiate back-to-origin rules.

    Origin URL

    Select https in the first box, enter www.example.com in the second box, and leave the third box empty.

  6. Click OK.

    The following items describe the access process after you configure the preceding rule:

    1. A requester requests https://examplebucket.oss-cn-hangzhou.aliyuncs.com/examplefolder/example.txt for the first time.

    2. If the examplefolder/example.txt object does not exist in examplebucket, OSS retrieves the object from the origin at https://www.example.com/examplefolder/example.txt.

    3. If the object is retrieved from the origin, OSS writes the example.txt object to the examplefolder directory in examplebucket and then returns the object to the requester. If the object is not retrieved from the origin, HTTP status code 404 is returned to the requester.

You can use the preceding steps to configure mirroring-based back-to-origin rules for basic business scenarios. For more information about mirroring-based back-origin rules for specific scenarios, see Configuration examples of mirroring-based back-to-origin.

Use OSS SDKs

The following sample code provides examples on how to configure mirroring-based back-to-origin rules by using OSS SDKs for common programming languages. For more information about how to configure mirroring-based back-to-origin rules by using OSS SDKs for other programming languages, see Overview.

import com.aliyun.oss.ClientException;
import com.aliyun.oss.OSS;
import com.aliyun.oss.common.auth.*;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.OSSException;
import com.aliyun.oss.model.RoutingRule;
import com.aliyun.oss.model.SetBucketWebsiteRequest;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class Demo {

    public static void main(String[] args) throws Exception {
        // In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
        String endpoint = "https://oss-cn-hangzhou.aliyuncs.com";
        // Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
        EnvironmentVariableCredentialsProvider credentialsProvider = CredentialsProviderFactory.newEnvironmentVariableCredentialsProvider();
        // Specify the name of the bucket. Example: examplebucket. 
        String bucketName = "examplebucket";

        // Create an OSSClient instance. 
        OSS ossClient = new OSSClientBuilder().build(endpoint, credentialsProvider);

        try {
            SetBucketWebsiteRequest request = new SetBucketWebsiteRequest(bucketName);
            // Specify the operation that you want to perform if the default homepage is specified, the name of the requested object does not end with a forward slash (/), and the object does not exist. 
            //request.setSubDirType(null);
            // Specify whether to redirect the requester to the default homepage in the subdirectory when the subdirectory is accessed. 
            //request.setSupportSubDir(false);

            List<RoutingRule> routingRules = new ArrayList<RoutingRule>();

            RoutingRule rule = new RoutingRule();
            rule.setNumber(1);
            // Specify the prefix that is contained in the object names. Only the objects whose names contain the specified prefix match the rule. 
            rule.getCondition().setKeyPrefixEquals("examplebucket");
            // Specify the HTTP status code. The rule is matched only when the specified object is accessed and HTTP status code 404 is returned. 
            rule.getCondition().setHttpErrorCodeReturnedEquals(404);
            // Specify the redirection type. 
            rule.getRedirect().setRedirectType(RoutingRule.RedirectType.Mirror);
            // Specify the URL of the origin for the mirroring-based back-to-origin rule. Example: https://www.example.com/. 
            rule.getRedirect().setMirrorURL("<yourMirrorURL>");
            //rule.getRedirect().setMirrorRole("AliyunOSSMirrorDefaultRole");
            // Specify whether to include the request parameters when the redirection rule or the mirroring-based back-to-origin rule is executed. 
            rule.getRedirect().setPassQueryString(true);
            // This parameter is used in the same manner as the PassQueryString parameter and is assigned a higher priority level than the PassQueryString parameter. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            rule.getRedirect().setMirrorPassQueryString(true);
            // Specify the HTTP status code in the response when the request is redirected. This parameter takes effect only when you set the RedirectType parameter to External or AliCDN. 
            //rule.getRedirect().setHttpRedirectCode(302);
            // Specify the domain name that you want to use for redirection. The domain name must comply with the naming conventions for domain names. 
            //rule.getRedirect().setHostName("oss.aliyuncs.com");
            // Specify the protocol that you want to use for redirection. This parameter takes effect only when you set the RedirectType parameter to External or AliCDN. 
            //rule.getRedirect().setProtocol(RoutingRule.Protocol.Https);
            // Specify the string that you want to use to replace the object name when the request is redirected. You can set this parameter to a variable. 
            //rule.getRedirect().setReplaceKeyWith("${key}.jpg");
            // If you set this parameter to true, the prefix of the object name is replaced with the value specified by the ReplaceKeyPrefixWith parameter. 
            rule.getRedirect().setEnableReplacePrefix(true);
            // Specify the string that you want to use to replace the prefix of the object name during redirection. 
            rule.getRedirect().setReplaceKeyPrefixWith("examplebucket");
            // Specify whether to check the MD5 hash of the response body that is returned by the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            rule.getRedirect().setMirrorCheckMd5(true);

            RoutingRule.MirrorHeaders mirrorHeaders = new RoutingRule.MirrorHeaders();
            // Specify whether to pass through all request headers except the following header to the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            mirrorHeaders.setPassAll(false);
            List passes = new ArrayList<String>();
            passes.add("cache-control");
            // Specify the headers that you want to pass through to the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            mirrorHeaders.setPass(passes);
            List removes = new ArrayList<String>();
            removes.add("content-type");
            // Specify the headers that you do not want to pass through to the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            mirrorHeaders.setRemove(removes);
            List sets = new ArrayList<Map<String, String>>();
            Map header1 = new HashMap<String, String>();
            header1.put("Key", "key1");
            header1.put("Value", "value1");
            Map header2 = new HashMap<String, String>();
            header2.put("Key", "key2");
            header2.put("Value", "value2");
            sets.add(header1);
            sets.add(header2);
            // Specify the headers that you want to pass through to the origin. The specified headers are passed through to the origin regardless of whether the headers are included in the request. 
            mirrorHeaders.setSet(sets);
            // Specify the headers that you want to include in the response when the requested object is obtained from the origin. This parameter takes effect only when you set the RedirectType parameter to Mirror. 
            rule.getRedirect().setMirrorHeaders(mirrorHeaders);

            routingRules.add(rule);
            request.setRoutingRules(routingRules);
            ossClient.setBucketWebsite(request);
        } catch (OSSException oe) {
            System.out.println("Caught an OSSException, which means your request made it to OSS, "
                    + "but was rejected with an error response for some reason.");
            System.out.println("Error Message:" + oe.getErrorMessage());
            System.out.println("Error Code:" + oe.getErrorCode());
            System.out.println("Request ID:" + oe.getRequestId());
            System.out.println("Host ID:" + oe.getHostId());
        } catch (ClientException ce) {
            System.out.println("Caught an ClientException, which means the client encountered "
                    + "a serious internal problem while trying to communicate with OSS, "
                    + "such as not being able to access the network.");
            System.out.println("Error Message:" + ce.getMessage());
        } finally {
            if (ossClient != null) {
                ossClient.shutdown();
            }
        }
    }
}
#-*-coding:utf-8-*-
import oss2
from oss2.models import BucketWebsite, MirrorHeadersSet, RedirectMirrorHeaders, Redirect, RoutingRule, \
    REDIRECT_TYPE_MIRROR, Condition
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# In this example, the endpoint of the China (Hangzhou) region is used. Specify your actual endpoint. 
# Specify the name of the bucket. Example: examplebucket. 
bucket = oss2.Bucket(auth, 'https://oss-cn-hangzhou.aliyuncs.com', 'examplebucket')

# Enable static website hosting, set the default homepage to index.html, and then set the default 404 page to error.html. 
index_file = 'index.html'
error_file = 'error.html'
# Specify the match conditions. 
condition1 = Condition(key_prefix_equals='examplefolder',
                       http_err_code_return_equals=404)

# Specify the headers that you want to include in the request when you use mirroring-based back-to-origin. 
mirror_headers_set_1 = MirrorHeadersSet("myheader-key5","myheader-value5")
mirror_headers_set_2 = MirrorHeadersSet("myheader-key6","myheader-value6")
set_list = [mirror_headers_set_1, mirror_headers_set_2]
pass_list = ['myheader-key1', 'myheader-key2']
remove_list = ['myheader-key3', 'myheader-key4']
mirror_header = RedirectMirrorHeaders(pass_all=True, pass_list=pass_list, remove_list=remove_list, set_list=set_list)
# Specify the operation to perform after the rule is matched. 
redirect1 = Redirect(redirect_type=REDIRECT_TYPE_MIRROR, mirror_url='https://www.example.com/',
                     mirror_pass_query_string=True, mirror_follow_redirect=True, mirror_check_md5=True, mirror_headers=mirror_header)

rule1 = RoutingRule(rule_num=1, condition=condition1, redirect=redirect1)
website_set = BucketWebsite(index_file, error_file, [rule1])

# Configure mirroring-based back-to-origin. 
bucket.put_bucket_website(website_set)
package main

import (
	"fmt"
	"github.com/aliyun/aliyun-oss-go-sdk/oss"
	"os"
)

func main() {
	// Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured. 
	provider, err := oss.NewEnvironmentVariableCredentialsProvider()
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Create an OSSClient instance. 
	// Specify the endpoint of the region in which the bucket is located. For example, if the bucket is located in the China (Hangzhou) region, set the endpoint to https://oss-cn-hangzhou.aliyuncs.com. Specify your actual endpoint. 
	client, err := oss.New("yourEndpoint", "", "", oss.SetCredentialsProvider(&provider))
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}

	// Specify the name of the bucket. Example: examplebucket. 
	bucketName := "examplebucket"

	var indexWebsite = "myindex.html"
	var errorWebsite = "myerror.html"

	btrue := true
	bfalse := false
	// Set the back-to-origin type to mirroring-based back-to-origin. 
	ruleOk := oss.RoutingRule{
		RuleNumber: 1,
		Condition: oss.Condition{
			KeyPrefixEquals: "",
			// Set the back-to-origin condition to HTTP status code 404. 
			HTTPErrorCodeReturnedEquals: 404,
		},
		Redirect: oss.Redirect{
			RedirectType: "Mirror",
			// PassQueryString: &btrue,
			// Specify the back-to-origin URL for a mirroring-based back-to-origin rule. 
			MirrorURL: "http://www.test.com/",
			// MirrorPassQueryString:&btrue,
			// MirrorFollowRedirect:&bfalse,
			// MirrorCheckMd5:&bfalse,
			MirrorHeaders: oss.MirrorHeaders{
				// PassAll:&bfalse,
				// Transmit specific HTTP headers. 
				Pass: []string{"myheader-key1", "myheader-key2"},
				// Prohibit the transmission of specific HTTP headers. 
				Remove: []string{"myheader-key3", "myheader-key4"},
				Set: []oss.MirrorHeaderSet{
					{
						Key:   "myheader-key5",
						Value: "myheader-value5",
					},
				},
			},
		},
	}

	// Set the back-to-origin type to redirection-based back-to-origin. 
	ruleArrOk := []oss.RoutingRule{
		{
			RuleNumber: 2,
			Condition: oss.Condition{
				// Set the back-to-origin condition to HTTP status code 404 and the prefix in object names to abc/. 
				KeyPrefixEquals:             "abc/",
				HTTPErrorCodeReturnedEquals: 404,
				IncludeHeader: []oss.IncludeHeader{
					{
						Key:    "host",
						Equals: "test.oss-cn-beijing-internal.aliyuncs.com",
					},
				},
			},
			Redirect: oss.Redirect{
				RedirectType:     "AliCDN",
				Protocol:         "http",
				HostName:         "www.test.com",
				PassQueryString:  &bfalse,
				ReplaceKeyWith:   "prefix/${key}.suffix",
				HttpRedirectCode: 301,
			},
		},
		// Set the back-to-origin type to mirroring-based back-to-origin. 
		{
			RuleNumber: 3,
			Condition: oss.Condition{
				KeyPrefixEquals:             "",
				HTTPErrorCodeReturnedEquals: 404,
			},
			Redirect: oss.Redirect{
				RedirectType:          "Mirror",
				PassQueryString:       &btrue,
				MirrorURL:             "http://www.test.com/",
				MirrorPassQueryString: &btrue,
				MirrorFollowRedirect:  &bfalse,
				MirrorCheckMd5:        &bfalse,
				MirrorHeaders: oss.MirrorHeaders{
					PassAll: &btrue,
					Pass:    []string{"myheader-key1", "myheader-key2"},
					Remove:  []string{"myheader-key3", "myheader-key4"},
					Set: []oss.MirrorHeaderSet{
						{
							Key:   "myheader-key5",
							Value: "myheader-value5",
						},
					},
				},
			},
		},
	}

	wxmlOne := oss.WebsiteXML{
		IndexDocument: oss.IndexDocument{
			Suffix: indexWebsite,
		},
		ErrorDocument: oss.ErrorDocument{
			Key: errorWebsite,
		},
	}
	wxmlOne.RoutingRules = append(wxmlOne.RoutingRules, ruleOk)
	wxmlOne.RoutingRules = append(wxmlOne.RoutingRules, ruleArrOk...)
	err = client.SetBucketWebsiteDetail(bucketName, wxmlOne)
	if err != nil {
		fmt.Println("Error:", err)
		os.Exit(-1)
	}
}

     

Use ossutil

You can use ossutil to configure mirroring-based back-to-origin rules. For more information, see Add or modify website-related configurations.

Use the OSS API

If your business requires a high level of customization, you can directly call RESTful APIs. To directly call an API, you need to include the signature calculation in your code. For more information, see PutBucketWebsite.

References

You can use the real-time log query feature to query objects uploaded to a bucket by using back-to-origin. For more information, see Real-time log query.