This topic describes how to use scheduling and overwrite policies to migrate incremental data.
Overview
When Data Online Migration runs a migration task, all data at the source data address is scanned in each execution of the migration task. If incremental data is generated during the migration, the incremental data is migrated to the destination data address after the data is scanned in the next execution. This topic describes how to use scheduling and overwrite policies to migrate incremental data.
Data Online Migration cannot ensure the timeliness of incremental data migrations. Therefore, this solution is not applicable to scenarios in which a large number of files are migrated and strict timeliness is required.
During the migration, data consistency cannot be ensured. Therefore, you must verify the migrated data at the destination data address after the migration task is complete. Before you delete the data at the source data address, make sure that the migrated data at the destination data address is correct. Otherwise, you are liable for the losses and consequences caused by any data loss.
Migration tutorials
You can use scheduling and overwrite policies to migrate incremental data in various scenarios. For more information, see Migration tutorials. View the migration tutorial that is applicable to your scenario.
When you create a migration task, you must configure the Execution Time parameter to specify a scheduling policy and configure the Overwrite Method parameter to specify an overwrite policy. The following table describes the policies.
Policy | Description |
Overwrite policy | Valid values of the Execution Time parameter: Do not overwrite, Overwrite All, and Overwrite based on the last modification time. To migrate incremental data, select Overwrite based on the last modification time. If a file at the destination data address meets specified conditions, the file is not overwritten. Note In this case, the system still needs to request the meta information of files from the source and destination data addresses. |
Scheduling policy | The migration task can be run within the specified time period or based on the execution frequency. For more information, see the "Supported execution frequencies" section of the Migrate data topic. Each time the migration task is executed, the execution ID increases by 1 until the task is run the specified number of times. The task is automatically scheduled based on the specified start time and pause time. The automatic scheduling stops after the task is run the specified number of times. However, you can still manually start the task. |
Data Online Migration automatically schedules the migration task based on the scheduling policy. In each execution of the migration task, full data is scanned. If a file already exists at the destination data address, the system determines whether to skip the file with the same name at the source data address based on the conditions of the overwrite policy that uses the last modification time. This way, incremental data migration is implemented.
For more information about how to run a migration task within the specified time period or based on the execution frequency, see the description of the Execution Time parameter in the "Create a migration task" section of the "Migrate data" topic in migration tutorials.
For more information about how to configure an overwrite policy, see the description of the Overwrite Method parameter in the "Create a migration task" section of the "Migrate data" topic in migration tutorials.
Example
Migrate incremental data at 00:00 every day
In this example, 2,000 historical data files are stored in the source Object Storage Service (OSS) bucket, and about 50 incremental data files are occasionally generated every day.
If you want to migrate the incremental data files to the destination OSS bucket at 00:00 every day, perform the following steps:
Create a migration task. Set the Overwrite Method parameter to Overwrite based on the last modification time, the Execution Time parameter to Periodic Scheduling, and the Execution Frequency parameter to Every Day, specify that the migration task runs at 00:00, and then set the Executions parameter to 100.
After the configurations are complete, Data Online Migration automatically starts the migration task at 00:00 every day, scans full data at the source data address, and then migrates incremental data to the destination data address based on the specified overwrite policy that uses the last modification time. The automatic scheduling stops after the migration task is run 100 times. However, you can still manually start the migration task.
If a large number of files need to be migrated and the migration task is not complete within a day, the task is still running when Data Online Migration starts the task at 00:00 the next day. In this case, Data Online Migration attempts to start the task at 00:00 on the day after the next day.