ApsaraDB for OceanBase:View the details of a data migration task

Schema migration

The definitions of data objects, such as tables, indexes, constraints, comments, and views, are migrated from the source database to the target database. Temporary tables are automatically filtered out. If the source database is not an OceanBase database, the data transmission service automatically performs data type conversion and SQL syntax concatenation based on the syntax definition of the mode of the target OceanBase Database tenant, and then replicates the data to the target database. If a migration object with the same name already exists in the target, the data transmission service will skip the migration object. You must ensure the consistency of table schemas between the source and the target.

If you have not manually dropped the hidden columns and unique indexes, the data transmission service will automatically drop the hidden columns and unique indexes based on the type of the data migration task in the forward switchover phase. For more information, see Hidden column mechanism of the data transmission service.

You can view the status, start time, completion time, total amount of time consumed, and table and view migration progress for a schema migration task on the Schema Migration page.

You can also perform the following operations on an object:

• View Creation Syntax: On the Database or table tab, click View in the Actions column of an object to view the database or table creation syntax.

  Compatible DDL syntax executed on the OBServer node is displayed. Incompatible syntax is converted before it is displayed.

• Modify Creation Syntax and Try Again: View the error information, check and modify the definition of the conversion result of a failed DDL statement, and then migrate the data to the target again.

• Retry/Retry All Failed Objects: You can retry failed schema migration tasks one by one or retry all failed tasks at a time.

• Skip/Batch Skip: You can skip failed schema migration tasks one by one or skip multiple failed tasks at a time. To skip multiple objects at a time, click Batch Skip in the upper-right corner.

• Remove/Batch Remove: You can remove failed schema migration tasks one by one or remove multiple failed tasks at a time. To remove multiple failed tasks at a time, click Batch Remove in the upper-right corner.

• View Database Return Code: You can view the DDL statements executed on the OBServer node and the execution error information of a failed schema migration task.

Full migration

The existing data is migrated from tables in the source database to the corresponding tables in the target database. On the Full Migration page, you can filter objects by source and target databases, or select View Objects with Errors to filter out objects that hinder the overall migration progress. You can also view related information on the Table Objects, Table Indexes, and Full Migration Performance tabs. The status of a full migration task changes to Completed only after both the table objects and table indexes are migrated.

• On the Table Objects tab, you can view the names, source and target databases, estimated data volume, migrated data volume, and status of tables.

• On the Table Indexes tab, you can view the objects, source and target databases, creation time, end time, amount of time consumed, and status of tables. You can also view the index creation syntax, retry a single failed object or all failed indexes, and skip or remove one or more failed indexes.

• On the Full Migration Performance tab, you can view the graphs of performance metrics such as the requests per second (RPS) and migration traffic of the source and target, average read time and slicing time at the source, average write time at the target, and performance benchmark, to identify performance issues in a timely manner.

  

  You can combine full migration with incremental synchronization to ensure data consistency between the source and target databases. If any objects fail to be migrated during full migration, the causes of the failure are displayed.

  Important

  If you did not select Schema Migration for Migration Type, the fields in the source database that match those in the target database are migrated during full migration. The system does not check whether the table structures are consistent.

Incremental synchronization

After incremental synchronization starts, the data transmission service synchronizes the data that has been changed (added, modified, or deleted) in the source database to the corresponding tables in the target database. When services continuously write data to the source database, the data transmission service starts the incremental data pull module to pull incremental data from the source instance, parses and encapsulates the incremental data, and then stores the data. After that, the data transmission service starts the full migration.

After the full migration task is completed, the data transmission service starts the incremental data replay module to pull incremental data from the incremental data pull module. The incremental data is synchronized to the target database after being filtered, mapped, and converted. If you execute a DDL statement in the source database and cause a JDBCWriter exception, which then causes the data migration task to fail, the DDL statement and the Skip button are displayed. You can click Skip and confirm your operation.

When you create a data migration task, we recommend that you enable alert monitoring to monitor the task status in real time. For more information, see Configure alert monitoring settings. If alert monitoring is disabled, when the incremental synchronization latency is less than or equal to 1 minute, the incremental synchronization status changes from Running to Monitoring by default.

Take note of the following items if alert monitoring is enabled:

• When the incremental synchronization latency exceeds the specified alert threshold, the incremental synchronization status stays at Running and the system does not trigger any alerts.

• When the incremental synchronization latency is less than or equal to the specified alert threshold, the incremental synchronization status changes from Running to Monitoring. After the incremental synchronization status changes to Monitoring, it will not change back to Running when the latency exceeds the specified alert threshold.

For a data migration task in the Running state, you can view its latency, current timestamp, and incremental synchronization performance in the incremental synchronization section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a data migration task in the Paused or Failed state, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled. You can also view the specific information about incremental synchronization objects and the incremental synchronization performance.

• The Synchronization Object Statistics tab displays the statistics about table-level DML statements executed for each incremental synchronization object in the current task. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Incremental Synchronization Objects tab are the sums of the corresponding columns on this tab.

  

  If you enable the DDL/DML statistic feature and then disable it, the data transmission service will record the historical statistics.

• The Incremental Synchronization Performance tab displays the following content:

  • Latency: the latency in synchronizing incremental data from the source database to the target database, in seconds.

  • Migration traffic: the traffic throughput of incremental data synchronization from the source database to the target database, in Kbit/s.

  • Average execution time: the average execution time of an SQL statement, in milliseconds.

  • Average commit time: the average commit time of a transaction, in milliseconds.

  • RPS: the number of records processed per second.

Full verification

After full migration and incremental synchronization are completed, the data transmission service automatically initiates a full verification task to verify the data tables in the source and target data sources.

The data transmission service also provides the corresponding APIs for you to initiate custom data verification tasks during incremental data synchronization.

On the Full Verification page, you can view the overall status, start time, end time, total amount of time consumed, estimated total number of rows, number of verified rows, RPS, and real-time traffic of the full verification task.

The Full Verification page contains the Verified Objects and Verification Performance tabs.

• On the Verified Objects tab, you can view the verification progress and verification object list.

  

  • You can view the names, source and target databases, full verification progress and results, and result summary of all migration objects.

  • You can filter migration objects by source database, target database, or verification result.

  • You can select View Completed Objects Only to view the basic information such as the names of objects for which schema migration has been completed, such as the object names.

  • You can choose Re-verify > Restart Full Verification to run a full verification again for all migration objects.

  • Observe the following notes for tables with inconsistent verification results:

    If you need to obtain the original inconsistent data to check or analyze the verification result, click Download inconsistent results to download the inconsistent data to your computer.

    Important

    You can download inconsistent data only when full verification is in the Completed state and data whose verification result is Inconsistent exists.

    If you need to reverify all data in the tables, choose Re-verify > Reverify Abnormal Table.

    If you only need to reverify inconsistent data, choose Re-verify > Verify Inconsistent Objects.

    Important

    Correction operations are not supported if the source database has no corresponding data.

• On the Full Verification Performance tab, you can view the graphs of RPS and migration traffic of the source and target databases, performance benchmark, and other performance statistics to identify performance issues in a timely manner.

The data transmission service allows you to skip full verification for a task that is being verified or has failed verification. On the Full Verification page, click Skip Full Verification. In the dialog box that appears, click OK.

After full verification is completed, you can click Go To Next Stage to start a forward switchover. After you enter the switchover process, you cannot rerun the current verification task to compare or correct data.

Forward switchover

Forward switchover is an abstract and standard process of traditional system cutover and does not involve the switchover of application connections. This process includes a series of tasks that are performed for application switchover in a data migration task. You must make sure that the entire forward switchover process is completed before the application connections are switched over to the target database.

Forward switchover is required for data migration. The data transmission service can ensure the completion of forward data migration in this process, and you can start the reverse incremental synchronization component based on your business needs. The forward switchover process involves the following operations:

1. You must make sure that data migration is completed and wait until forward synchronization is completed.

2. The data transmission service automatically supplements the check constraints and FOREIGN KEY constraints that are ignored in schema migration when you migrate data to an Oracle database or an Oracle tenant of OceanBase Database.

3. The data transmission service automatically deletes the hidden columns and unique indexes on which the migration depends.

   This operation is performed only for data migration between an Oracle database and an OceanBase database or between OceanBase databases. For more information, see Hidden column mechanisms.

4. You must migrate triggers, functions, and stored procedures in the source database that are not supported by the data transmission service to the target database.

5. You must disable triggers and FOREIGN KEY constraints in the source database. This operation is required only when the data migration task involves reverse incremental synchronization.

The forward switchover process contains the following steps:

1. Start forward switchover.

   In this step, you can start forward switchover, but no operation is performed in the background. If you make sure that data migration is completed, you can click Start Forward Switchover to start the process.

   Important

   Before you start forward switchover, make sure that data writing has stopped in the source database.

2. Perform a switchover precheck.

   In this step, the data transmission service checks the following items:

   • Synchronization latency between the source and target databases. If the synchronization latency is within 15 seconds, this check item is passed.

   • Write privilege of the account in the source database. If the data migration task involves reverse incremental synchronization, the data transmission service additionally checks whether the account configured in the source database has the privilege to write data, to ensure that data can be properly written during reverse incremental synchronization.

   • Privilege of the account in the target database for incremental data read. If the data migration task involves reverse incremental synchronization, the data transmission service additionally checks whether the account configured in the target database has the privilege to read data. This ensures that data can be properly written to the target database during reverse incremental synchronization.

   • Incremental logs in the target database. If the data migration task involves reverse incremental synchronization, the data transmission service additionally checks whether the incremental logging configuration in the target database meets the log extraction requirements of reverse incremental synchronization.

   If the switchover precheck is passed, the data transmission service automatically performs the next step. If the precheck fails, you can click Retry or Skip.

   Important

   If you click Skip, data loss may occur in the target database, or the reverse incremental synchronization process may fail. Proceed with caution.

3. Start the target store.

   Note

   This step is available only when the data migration task involves reverse incremental synchronization.

   If the precheck for forward switchover is passed, the data transmission service automatically starts incremental log pulling for the target database. This way, the data transmission service obtains DML and DDL operations in the target database and parse and save related log data to prepare for reverse incremental synchronization. This step takes 3 to 5 minutes.

4. Confirm that data writing has stopped in the source database.

   In this step, the data transmission service checks whether business data is still being written to the source database. After you confirm that no new data is written to the source database, click OK to go to the next step.

5. Confirm the data writing stop timestamp upon synchronization completion.

   In this step, the data transmission service checks whether the target database is synchronized to the data writing stop timestamp in the source database. If this step is in progress or fails and the synchronization is not completed after a long period, you can click Skip.

   Important

   If you choose to skip this step, data inconsistency may occur between the source and target. Proceed with caution.

6. Stop forward synchronization.

   In this step, you can stop forward synchronization. After forward synchronization is stopped, any data changes in the source database will no longer be synchronized to the target database. If forward synchronization fails to be stopped, you can click Retry or Skip.

   Important

   You can click Skip only after you confirm that forward synchronization is completed in the background. Otherwise, data in the source database may be unexpectedly written to the target database. Proceed with caution.

7. Process database objects.

   In this step, you can process the objects that are ignored in data migration or not supported by the data transmission service, to ensure normal operation of your business after the switchover to the target database. The following operations are supported:

   • Migrate database objects to the target database: You must migrate triggers, functions, and stored procedures in the source database that are not supported by the data transmission service to the target database. After you complete the migration, click Mark as Complete.

   • Disable triggers and FOREIGN KEY constraints in the source database: This operation is required only when the data migration task involves reverse incremental synchronization. It prevents data from being affected by triggers or FOREIGN KEY constraints, to avoid failures of reverse incremental synchronization. After you complete this operation, click Mark as Complete.

   • Supplement the objects ignored in schema migration to the target database: If the target is an Oracle database or an Oracle tenant of OceanBase Database, the data transmission service automatically supplements the objects that are ignored in schema migration, such as CHECK constraints and FOREIGN KEY constraints, to the target. In other cases, the preceding objects are automatically migrated during schema migration, and no extra operation is required.

   • Drop hidden columns and unique indexes added by the data transmission service: This operation is required only when you migrate data between an Oracle database and an OceanBase database or between OceanBase databases. The data transmission service automatically drops the hidden columns and unique indexes that are added to the target database to ensure data consistency. This operation runs automatically, and the amount of time required depends on the amount of data in the target database. You can click Skip to skip this operation, but you must manually perform it later. Proceed with caution. For more information, see Mechanisms for handling hidden columns.

8. Start reverse incremental synchronization.

   Note

   This step is available only when the data migration task involves reverse incremental synchronization.

   In this step, you can start incremental synchronization for the target database to synchronize incremental DML or DDL operations from the target database to the source database in real time. The configuration of incremental synchronization is the same as that specified when the task was created. For more information, see Incremental synchronization of DDL operations.

Reverse incremental synchronization

For a Running data migration task, you can view its latency, current timestamp, and performance of reverse incremental synchronization in the Reverse Incremental Migration section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a data migration task in the Paused or Failed state, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled. You can also view the specific information about the objects and performance of reverse incremental synchronization.

• The Synchronization Object Statistics tab displays the statistics on table-level DML statements executed for each incremental synchronization object in the current task. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Synchronization Object Statistics tab are the sums of the statistics in the corresponding columns on this tab.

• The Reverse Incremental Migration Performance tab displays the following content:

  • Latency: the latency in synchronizing incremental data from the target database to the source database, in seconds.

  • Migration traffic: the traffic throughput of incremental data synchronization from the target database to the source database, in Kbit/s.

  • Average execution time: the average execution time of an SQL statement, in milliseconds.

  • Average commit time: the average commit time of a transaction, in milliseconds.

  • RPS: the number of records processed per second.