Data Online Migration:Migrate data

Usage notes

When you migrate data by using Data Online Migration, take note of the following items:

• Data Online Migration accesses the source data address by using the public interfaces provided by the storage service provider of the source data address. The access behavior depends on the interface implementation of the storage service provider.

• When Data Online Migration is used for migration, it consumes resources at the source and the destination data addresses. This may interrupt your business. To ensure business continuity, we recommend that you enable throttling for your migration tasks or run the migration tasks during off-peak hours after careful assessment.

• Before a migration task starts, Data Online Migration checks the files at the source and the destination data addresses. If a file at the source data address and a file at the destination data address have the same name, and the File Overwrite Method parameter of the migration task is set to Yes, the file at the destination data address is overwritten during migration. If the two files contain different information and the file at the destination data address needs to be retained, we recommend that you change the name of one file or back up the file at the destination data address.

• The LastModified property of the source file is retained after the file is migrated to the destination bucket. If a lifecycle rule is configured for the destination bucket and takes effect, the migrated file whose last modification time is within the specified time period of the lifecycle rule may be deleted or archived in specific storage types.

Limits

Data Online Migration allows you to migrate only the data of a single bucket per task. You cannot migrate all data that belongs to your account in a single task.

Step 1: Create a source data address

1. Log on to the Data Online Migration console as the Resource Access Management (RAM) user that you created for data migration.

2. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.

3. In the Create Address panel, configure the parameters and click OK. The following table describes the parameters.

   Parameter	Required	Description
   Data Type	Yes	The type of the source data address. Select YOU PAI.
   Data Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Domain Address	Yes	The endpoint of an OSS SDK or the domain name in a requested URL when you call a RESTful API operation. For more information, see REST API in the USS official documentation. Important The information in the preceding link may be outdated due to changes in the original server and is for reference only.
   Use CDN	No	Specify whether to use a CDN domain to migrate data. Do not use: A CDN domain is not used. The endpoint that is specified in the Domain Address parameter is used to migrate data. Use: Enter the CDN domain name that is configured for your bucket. The specified CDN domain is used to migrate data in the migration task. Important You can use a CDN domain to increase the migration speed and minimize the impact of traffic throttling that is implemented by the UPYUN origin. When you use a CDN domain to migrate data, you may migrate expired data. You must verify data after the migration task is complete. If you want to use an HTTPS-based domain name, you must enable the HTTPS protocol for the UPYUN CDN service. For more information, see How do I configure HTTPS? in the USS official documentation. This link is for reference only and may be outdated due to changes to the origin server.
   Service Name	Yes	You can log on to the UPYUN Cloud console to view the service name on the UPYUN Storage Service page.
   Migration Folder	Yes	The prefix of the source data address. You can specify a prefix to migrate specific data. If you specify a prefix, only the data in the directory that is specified by the prefix is migrated. A prefix cannot start with a forward slash (/) and must end with a forward slash (/). If you do not specify a prefix, all data in the Amazon S3 bucket is migrated.
   Operator Name	Yes	Log on to the UPYUN Cloud console. On the Account Management page, click Create Operator to create an operator for data migration. You must grant the read permission to the operator. On the UPYUN Storage Service page, click the name of a bucket. On the configuration page, select Content Management and click Authorize to authorize the operator. Important The console link is for reference only and may be outdated due to changes to the origin server.
   Operator Secret	Yes

Step 2: Create a destination data address

1. In the left-side navigation pane, choose Data Online Migration > Data Address. On the page that appears, click Create Data Address.

2. In the Create Data Address panel, configure the parameters and click OK. The following table describes the parameters.

   Parameter	Required	Description
   Data Type	Yes	The type of the destination data address. Select OSS.
   Data Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Data Region	Yes	The region in which the destination data address resides.
   Enable Transfer Acceleration	No	If you want to use Data Online Migration together with the transfer acceleration feature of Object Storage Service (OSS), enable the transfer acceleration feature for buckets. The transfer acceleration feature takes effect within 30 minutes after it is enabled. After you enable the feature, wait for 30 minutes before you create a migration job. Important If you use the transfer acceleration feature to access your OSS buckets, you are charged additional transfer acceleration fees. For more information, see Transfer acceleration.
   OSS Endpoint	Yes	The endpoint of the destination data address. Select an endpoint based on the region in which the destination data address resides. For more information, see Regions and endpoints. Note If you want to migrate data from third-party cloud services to OSS, you must access OSS from a public endpoint over the Internet.
   AccessKey Id	Yes	The AccessKey pair of the RAM user that is used to write data to the destination data address. The AccessKey pair is used by OSS to check whether the RAM user has the permissions to write data to the destination data address.
   AccessKey Secret	Yes
   OSS Bucket	Yes	The OSS bucket to which the data is migrated.
   OSS Prefix	No	The prefix of the destination data address. If you specify a prefix, the source data is migrated to the specified directory. The value cannot start with a forward slash (/) and must end with a forward slash (/). Example: data/to/oss/. If you do not specify a prefix, the source data is migrated to the root directory of the destination bucket. Important If the name of a source file starts with a forward slash (/), you must specify an OSS prefix when you configure the destination data address. If no OSS prefix is specified, the migration job fails. For example, if the name of a file to be migrated is /test/test.png, you must specify an OSS prefix, such as oss/. After the migration task is complete, the name of the OSS file changes from /test/test.png to oss//test/test.png.

Step 3: Create a migration task

1. In the left-side navigation pane, choose Data Online Migration > Migration Jobs. On the page that appears, click Create Job.

2. In the Create Job panel, read the terms of the migration service and select I agree to the above terms and activate the Data Transport service. Then, click Next.

3. In the Fee Reminder message, click Yes, Go Ahead.

4. In the Job Config step, configure the parameters and click Next. The following table describes the parameters.

   Parameter	Required	Description
   Job Name	Yes	The name of the source data address. The name must be 3 to 63 characters in length. The name cannot contain special characters except for hyphens (-) and underscores (_).
   Source Data Address	Yes	The source data address that you created.
   Destination Data Address	Yes	The destination data address that you created.
   Specified Directory	No	The directories in which the files and subdirectories are migrated or not migrated. Valid values: Do not filter: migrates all data from the source data address. Exclude: does not migrate the files or subdirectories in the specified directories. For example, you want to migrate all the directories in the root_dir/ directory except root_dir/bad_sub1/ and root_dir/bad_sub2/. In this case, you can select this method and specify bad_sub1/ and bad_sub2/. Contain: migrates only the files and subdirectories in the specified directories. For example, you want to migrate only root_dir/good_sub1/ and root_dir/good_sub2/ in the root_dir/ directory. In this case, you can select this method and specify good_sub1/ and good_sub2/. Note A directory name can contain only digits and letters. If you use special characters, the migration job may fail. A directory name cannot start with a forward slash (/) or a backslash (\), and cannot contain double forward slashes (//), double periods (..), or double quotation marks ("). The character string that consists of all the specified directory names cannot exceed 10 KB in length. A directory name must end with a forward slash (/). For example, you can specify docs/ as the directory name. You can specify up to 20 directories of the Exclude or Contain type.
   Migration Type	Yes	The data migration mode. Valid values: Full: The system migrates all data from the source data address to the destination data address. You must configure the Start Time Point of File parameter. The system migrates all data whose last modification time is later than the specified time. After all data is migrated, the migration job ends. If you change the data at the source data address after the migration job is complete, you can submit another job to migrate all data. In this case, the system migrates only the changed data. Incremental: The system runs a migration job based on the values that you specified for the Migration Interval and Migration Times parameters. You must configure the Start Time Point of File parameter. During the first migration, the system migrates all data whose last modification time is later than the specified time. After the first migration is complete, incremental migration is performed based on the value that you specified for the Migration Interval parameter. For each incremental migration, the system migrates only the data that is created or modified after the previous migration starts and before the current migration starts. If you set the Migration Times parameter to N, full migration is performed once and then incremental migration is performed (N - 1) times. For example, the current time is 08:00, March 10, 2019 and you set the Migration Interval parameter to 1, the Migration Times parameter to 5, and the Start Time Point of File parameter to 08:00, March 5, 2019. When the first migration starts, the system migrates all files whose last modification time is between 08:00, March 5, 2019 and 08:00, March 10, 2019. In this example, the first migration requires one hour to complete and the second migration starts at 10:00, March 10, 2019. The duration from 08:00 to 10:00 consists of one hour for the first migration process and one hour for the migration interval. During the second migration, the system migrates the files whose last modification time is between 08:00, March 10, 2019 and 10:00, March 10, 2019. The migration job consists of one full migration and four incremental migrations. Important Before full or incremental migration starts, the system compares the files at the source data address with the files at the destination data address. If a source file has the same name as a destination file, the destination file is overwritten when one of the following conditions is met: The Content-Type values of the files at the source data address and the destination data address are different. The last modification time of the file at the source data address is later than that of the file at the destination data address. The size of the file at the source data address is different from that of the file at the destination data address.
   Start Time Point of File	Yes	The time to filter data to be migrated. The system migrates the data that is created or modified after the specified time. Valid values: All: migrates all files. Assign: migrates only the files that are created or modified after the specified time. For example, if you set the time to 08:00:00, November 1, 2018, only the files that are created or modified after 08:00:00, November 1, 2018 are migrated.
   Migration Interval	This parameter is required if you set the Migration Type parameter to Incremental.	The default value is 1, and the maximum value is 24. Unit: hours.
   Migration Times	This parameter is required if you set the Migration Type parameter to Incremental.	The default value is 1, and the maximum value is 30.
   File Overwrite Method	Yes	The method that is used to process the file at the source data address whose name is the same as that of a file at the destination data address. Valid values: LastModified: If a source file has the same name as a destination file, the system compares the last modified time indicated by the LastModified property of the files. If the last modification time of the file at the source data address is later than that of the file at the destination data address, the file at the source data address is migrated and the file at the destination data address is overwritten. If the last modification time of the file at the source data address is earlier than that of the file at the destination data address, the file at the source data address is not migrated and the file at the destination data address is retained. If the two files have the same last modification time, the system checks the values of the Size and Content-Type properties of the files. If the two files have the same Size and Content-Type values, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten. Condition: If a source file has the same name as a destination file, the system compares the values of the LastModified, Size, and Content-Type properties of the files. If all property values of the two files are the same, the file at the source data address is not migrated. If the two files have different values of at least one property, the file at the source data address is migrated and the file at the destination data address is overwritten. All: If a source file has the same name as a destination file, the system performs no comparison and directly overwrites the destination file with the source file. No: If a source file has the same name as a destination file, the system performs no comparison and directly skips the source file in the migration. Warning If you select Condition or LastModified, earlier files may overwrite later files. If you select Condition or LastModified, make sure that the source files have LastModified, Size, and Content-Type values. Otherwise, the overwrite policy may become ineffective and data is not migrated as expected.

5. In the Data Prediction section of the Performance step, configure the Data Size and File Count parameters.

   Note

   To ensure a successful migration job, we recommend that you estimate the volume of data that you want to migrate as accurately as possible. For more information, see the Estimate the amount of data that you want to migrate section of the "Preparations" topic.

6. Optional. In the Flow Control section of the Performance step, configure the Time Range and Max Flow(MB/s) parameters and click Add.

   Note

   To ensure business continuity, we recommend that you configure the Time Range and Max Flow(MB/s) parameters based on the workload peaks and troughs.

7. Click Create and wait until the migration job is complete.

Step 4: Verify data