ApsaraDB for OceanBase:View the details of a data migration project

Schema migration

The definitions of data objects, such as tables, indexes, constraints, comments, and views, are migrated from the source database to the destination database. Temporary tables are automatically filtered out. If the source database is not an OceanBase database, the data transmission service performs format conversion and encapsulation based on the syntax definition and standard of the type of the destination tenant of OceanBase Database and then replicates the data to the destination database. If a migration object with the same name already exists in the destination, the data transmission service will skip the migration object. You must ensure the consistency of table schemas between the source and the destination.

When you advance to the forward switchover phase in a data migration project, the hidden columns and unique indexes will be automatically dropped based on the type of the data migration project. For more information, see Hidden column mechanisms.

You can view the status, start time, completion time, total amount of time consumed, and table and view migration progress for a schema migration project on the Schema Migration page.

You can also perform the following operations on an object:

• View Creation Syntax: On the Database or table tab, click View in the Actions column of an object to view the database or table creation syntax.

  Compatible DDL syntax executed on the OBServer node is displayed. Incompatible syntax is converted before it is displayed.

• Modify Creation Syntax and Try Again: View the error information, check and modify the definition of the conversion result of a failed DDL statement, and then migrate the data to the destination again.

• Retry/Retry All Failed Objects: You can retry failed schema migration tasks one by one or retry all failed tasks at a time.

• Skip/Batch Skip: You can skip failed schema migration tasks one by one or skip multiple failed tasks at a time. To skip multiple objects at a time, click Batch Skip in the upper-right corner.

• Remove/Batch Remove: You can remove failed schema migration tasks one by one or remove multiple failed tasks at a time. To remove multiple failed tasks at a time, click Batch Remove in the upper-right corner.

• View Database Return Code: You can view the DDL statements executed on the OBServer node and the execution error information of a failed schema migration task.

Full migration

The existing data is migrated from tables in the source database to the corresponding tables in the destination database. On the Full Migration page, you can filter objects by source and destination databases, or select View Objects with Errors to filter out objects that hinder the overall migration progress. You can also view related information on the Table Objects, Table Indexes, and Full Migration Performance tabs. The status of a full migration task changes to Completed only after the table objects and table indexes are migrated.

• On the Table Objects tab, you can view the names, source and destination databases, estimated data volume, migrated data volume, and status of tables.

• On the Table Indexes tab, you can view the objects, source and destination databases, creation time, end time, amount of time consumed, and status of tables. You can also view the index creation syntax, retry a single failed object or all failed indexes, and skip or remove one or more failed indexes.

• On the Full Migration Performance tab, you can view the graphs of performance metrics such as the requests per second (RPS) and migration traffic of the source and destination, average read time and slicing time at the source, average write time at the destination, and performance benchmark, to identify performance issues in a timely manner.

  

  You can combine full migration with incremental synchronization to ensure data consistency between the source and destination databases. If any objects fail to be migrated during full migration, the causes of the failure are displayed.

  Important

  If you did not select Schema Migration for Migration Type, the fields in the source database that match those in the destination database are migrated during full migration. The system does not check whether the table structures are consistent.

Incremental synchronization

After incremental synchronization starts, the data transmission service synchronizes the data that has been changed (added, modified, or deleted) in the source database to the corresponding tables in the destination database. When services continuously write data to the source database, the data transmission service starts the incremental data pull module to pull incremental data from the source instance, parses and encapsulates the incremental data, and then stores the data. After that, the data transmission service starts the full data migration.

After the full data migration task is completed, the data transmission service starts the incremental data replay module to pull incremental data from the incremental data pull module. The incremental data is synchronized to the destination instance after being filtered, mapped, and converted. If you execute a DDL statement in the source database and cause a JDBCWriter exception, which then causes the data migration project to fail, the DDL statement and the Skip button are displayed. You can click Skip and confirm your operation.

When you create a data migration project, we recommend that you enable alert monitoring to monitor the project status in real time. For more information, see Configure alerts. If alert monitoring is disabled, when the incremental synchronization latency is less than or equal to 1 minute, the incremental synchronization status changes from Running to Monitoring by default.

Take note of the following items if alert monitoring is enabled:

• When the incremental synchronization latency exceeds the specified alert threshold, the incremental synchronization status stays at Running and the system does not trigger any alerts.

• When the incremental synchronization latency is less than or equal to the specified alert threshold, the incremental synchronization status changes from Running to Monitoring. After the incremental synchronization status changes to Monitoring, it will not change back to Running when the latency exceeds the specified alert threshold.

For a Running data migration project, you can view its latency, current timestamp, and incremental synchronization performance in the incremental synchronization section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a Paused or Failed data migration project, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled. You can also view the specific information about incremental synchronization objects and the incremental synchronization performance.

• The Synchronization Object Statistics tab displays the statistics about table-level DML statements executed for each incremental synchronization object in the current project. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Incremental Synchronization Objects tab are the sum of the corresponding columns on this tab.

  

  If you enable the DDL/DML statistic feature and then disable it, the data transmission service will record the historical statistics.

• The Incremental Synchronization Performance tab displays the following content:

  • Latency: the latency in synchronizing incremental data from the source database to the destination database, in seconds.

  • Migration traffic: the traffic throughput of incremental data synchronization from the source database to the destination database, in Kbit/s.

  • Average execution time: the average execution time of an SQL statement, in milliseconds.

  • Average commit time: the average commit time of a transaction, in milliseconds.

  • RPS: the number of records processed per second.

Full verification

After full migration and incremental synchronization are completed, the data transmission service automatically initiates a full verification task to verify the data tables in the source and destination data sources.

The data transmission service also provides the corresponding APIs for you to initiate custom data verification tasks during incremental data synchronization.

On the Full Verification page, you can view the overall status, start time, end time, total amount of time consumed, estimated total number of rows, number of verified rows, RPS, and real-time traffic of the full verification task.

The Full Verification page contains the Verified Objects and Verification Performance tabs.

• On the Verified Objects tab, you can view the verification progress and verification object list.

  

  • You can view the names, source and destination databases, full verification progress and results, and result summary of all migration objects.

  • You can filter migration objects by source database, destination database, or verification result.

  • You can select View Completed Objects Only to view the basic information of objects that have completed schema migration, such as the object names.

  • You can choose Re-verify > Restart Full Verification to run a full verification again for all migration objects.

  • Take note of the following items for tables with inconsistent verification results:

    If you need to obtain the original inconsistent data to check or analyze the verification result, click Download inconsistent results to download the inconsistent data to your computer.

    Important

    You can download inconsistent data only when full verification is in the Completed state and data whose verification result is Inconsistent exists.

    If you need to reverify all data in the tables, choose Re-verify > Reverify Abnormal Table.

    If you only need to reverify inconsistent data, choose Re-verify > Verify Inconsistent Objects.

    Important

    Correction operations are not supported if the source database has no corresponding data.

• On the Verification Performance tab, you can view the graphs of RPS and migration traffic of the source and destination databases, performance benchmark, and other performance statistics to identify performance issues in a timely manner.

The data transmission service allows you to skip full verification for a project that is being verified or has failed verification. On the Full Verification page, click Skip Full Verification. In the dialog box that appears, click OK.

After the full verification is completed, you can click Go To Next Stage to start a forward switchover. After you enter the switchover process, you cannot rerun the current verification task to compare or correct data.

Forward switchover

Forward switchover is an abstract and standard process of traditional system cutover and does not involve the switchover of application connections. This process includes a series of tasks that are performed for application switchover in a data migration project. You must make sure that the entire forward switchover process is completed before the application connections are switched over to the destination database.

Forward switchover is required for data migration. The data transmission service can ensure the completion of forward data migration in this process, and you can start the reverse incremental migration component based on your business needs. The forward switchover process involves the following operations:

1. You must make sure that data migration is completed and wait until forward synchronization is completed.

2. The data transmission service automatically supplements the check constraints and FOREIGN KEY constraints that are ignored in schema migration when you migrate data to an Oracle database or an Oracle tenant of OceanBase Database.

3. The data transmission service automatically deletes the hidden columns and unique indexes on which the migration depends.

   This operation is performed only for data migration between an Oracle database and an OceanBase database or between OceanBase databases. For more information, see Hidden column mechanisms.

4. You must migrate triggers, functions, and stored procedures in the source database that are not supported by the data transmission service to the destination database.

5. You must disable triggers and FOREIGN KEY constraints in the source database. This operation is required only when the data migration project involves reverse incremental migration.

The forward switchover process contains the following steps:

1. Start forward switchover.

   In this step, you can start forward switchover, but no operation is performed in the background. If you make sure that data migration is completed, you can click Start Forward Switchover to start the process.

   Important

   Before you start forward switchover, make sure that data writing has stopped in the source database.

2. Perform a switchover precheck.

   In this step, the data transmission service checks the following items:

   • Synchronization latency between the source and destination databases. If the synchronization latency is within 15 seconds, this check item is passed.

   • Write privilege of the account in the source database. If the data migration project involves reverse incremental migration, the data transmission service additionally checks whether the account configured in the source database has the privilege to write data, to ensure that data can be properly written during reverse incremental migration.

   • Privilege of the account in the destination database for incremental data read. If the data migration project involves reverse incremental migration, the data transmission service additionally checks whether the account configured in the destination database has the privilege to read data. This ensures that data can be properly written to the destination database during reverse incremental migration.

   • Incremental logs in the destination database. If the data migration project involves reverse incremental migration, the data transmission service additionally checks whether the incremental logging configuration in the destination database meets the log extraction requirements of reverse incremental migration.

   If the switchover precheck is passed, the data transmission service automatically performs the next step. If the precheck fails, you can click Retry or Skip.

   Important

   If you click Skip, data loss may occur in the destination database, or the reverse incremental migration process may fail. Proceed with caution.

3. Start the destination store.

   Note

   This step is available only when the data migration project involves reverse incremental migration.

   If the precheck for forward switchover is passed, the data transmission service automatically starts incremental log pulling for the destination database. This way, the data transmission service obtains DML and DDL operations in the destination database and parse and save related log data to prepare for reverse incremental migration. This step takes 3 to 5 minutes.

4. Confirm that data writing has stopped in the source database.

   In this step, the data transmission service checks whether business data is still being written to the source database. If you make sure that no new data is written to the source database, click OK to go to the next step.

5. Confirm the data writing stop timestamp upon synchronization completion.

   In this step, the data transmission service checks whether the destination database is synchronized to the data writing stop timestamp in the source database. If this step is in progress or fails and the synchronization is not completed after a long period, you can click Skip.

   Important

   If you choose to skip this step, data inconsistency may occur between the source and destination. Proceed with caution.

6. Stop forward synchronization.

   In this step, you can stop forward synchronization. After forward synchronization is stopped, any database changes in the source database will no longer be synchronized to the destination database. If forward synchronization fails to be stopped, you can click Retry or Skip.

   Important

   You can click Skip only when you make sure that forward synchronization is completed in the background. Otherwise, data in the source database may be unexpectedly written to the destination database. Proceed with caution.

7. Process database objects.

   In this step, you can process the objects that are ignored in data migration or not supported by the data transmission service, to ensure normal operation of your business after the switchover to the destination database. The following operations are supported:

   • Migrate database objects to the destination database: You must migrate triggers, functions, and stored procedures in the source database that are not supported by the data transmission service to the destination database. After you complete the migration, click Mark as Complete.

   • Disable triggers and FOREIGN KEY constraints in the source database: This operation is required only when the data migration project involves reverse incremental migration. It prevents data from being affected by triggers or FOREIGN KEY constraints, to avoid failures of reverse incremental migration. After you complete this operation, click Mark as Complete.

   • Supplement the objects ignored in schema migration to the destination database: If the destination is an Oracle database or an Oracle tenant of OceanBase Database, the data transmission service automatically supplements the objects that are ignored in schema migration, such as CHECK constraints and FOREIGN KEY constraints, to the destination. In other cases, the preceding objects are automatically migrated during schema migration, and no extra operation is required.

   • Delete hidden columns and unique indexes added by the data transmission service: This operation is required only when you migrate data between an Oracle database and an OceanBase database or between OceanBase databases. The data transmission service automatically deletes the hidden columns and unique indexes that are added to the destination database to ensure data consistency. This operation runs automatically, and the amount of time required depends on the amount of data in the destination database. You can click Skip to skip this operation, but you must manually perform it later. Proceed with caution. For more information, see Mechanisms for handling hidden columns.

8. Start reverse incremental migration.

   Note

   This step is available only when the data migration project involves reverse incremental migration.

   In this step, you can start incremental synchronization for the destination database to synchronize incremental DML or DDL operations from the destination database to the source database in real time. The configuration of incremental synchronization is the same as that specified when the project was created. For more information, see Incremental synchronization of DDL operations.

Reverse incremental synchronization

For a Running data migration project, you can view its latency, current timestamp, and performance of reverse incremental migration in the Reverse Incremental Migration section. The latency is displayed in the following format: X seconds (updated Y seconds ago). Normally, Y is less than 20.

For a Paused or Failed data migration project, you can enable the DDL/DML statistics feature to collect statistics on the database operations performed after this feature is enabled. You can also view the specific information about the objects and performance of reverse incremental migration.

• The Synchronization Object Statistics tab displays the statistics about table-level DML statements executed for each incremental synchronization object in the current project. The numbers displayed in the Change Sum, Delete, Insert, and Update fields in the section above the Synchronization Object Statistics tab are the sums of the corresponding columns on this tab.

• The Reverse Incremental Migration Performance tab displays the following content:

  • Latency: the latency in synchronizing incremental data from the destination database to the source database, in seconds.

  • Migration traffic: the traffic throughput of incremental data synchronization from the destination database to the source database, in Kbit/s.

  • Average execution time: the average execution time of an SQL statement, in milliseconds.

  • Average commit time: the average commit time of a transaction, in milliseconds.

  • RPS: the number of records processed per second.