DataWorks:Troubleshoot errors that occur on a full and incremental synchronization task used to synchronize data to MaxCompute

Common scenarios in which errors may occur on a full and incremental synchronization task used to synchronize data to MaxCompute

Scenarios in which no binary logs are lost

• The real-time synchronization subtask generated by the full and incremental synchronization task fails because data changes that are caused by specific DDL operations on source tables cannot be synchronized to the destination.

  Solutions:

  • Go to the Nodes page of Data Integration in the DataWorks console, find the full and incremental synchronization task, click More in the Actions column, and then select Edit. On the configuration page of the full and incremental synchronization task, remove the source tables from the task, save the modification, and then submit the task for running. Go to the configuration page of the real-time synchronization subtask in the same way, add the removed source tables to the task, and then submit the task for running. This way, the full and incremental synchronization task can synchronize data from the source tables again, and the DDL operations that are performed on the source tables can be ignored.

  • Before you start the real-time synchronization subtask, modify the DDL processing policies configured for the subtask. You can change the processing policies for the related DDL messages to ignoring or error reporting, as shown in the following figure.

    Note

    The modification to the DDL processing policies is a temporary operation. When the full and incremental synchronization task is rerun, the modification to the DDL processing policies configured for the real-time synchronization subtask is overwritten. You can also directly modify the DDL processing policies configured for the full and incremental synchronization task.

    

• The real-time synchronization subtask generated by the full and incremental synchronization task fails because dirty data is generated during data synchronization.

  • Dirty data may be generated because the data types of destination fields are incompatible with the data types of source fields. If the incompatibility occurs, view the run logs generated for the real-time synchronization subtask or the output information about the dirty data to determine whether the issue exists in the source or destination. If the issue exists in the schemas of destination tables, manually recreate destination tables to recover the real-time synchronization subtask.

  • Manually recreate destination tables to ensure that the data types of fields in the destination tables are compatible with the data types of fields in the source tables, remove the source tables in which the data types of fields are incompatible with the data types of fields in destination tables from the full and incremental synchronization task, add the source tables to the task again, and then change the table generation method of destination tables to Use Existing Table for the full and incremental synchronization task. Finally, run the full and incremental synchronization task to recover the real-time synchronization subtask.

• The real-time synchronization subtask generated by the synchronization task fails because an error occurs on the Tunnel service.

  You can view the logs generated for the real-time synchronization subtask to check whether the logs contain the InternalServerError and Exception when calling callable. Exception Msg: Read timed out error messages, as shown in the following figure. These error messages indicate that an issue occurs on the Tunnel service. If the logs contain the error messages, contact the on-duty engineers of MaxCompute to resolve the issue. After the issue is resolved, restart the real-time synchronization subtask.

Scenarios in which binary logs are lost

If binary logs are lost, some incremental data of the source cannot be synchronized to the destination. In this case, you must re-synchronize full data and incremental data from the source to the destination. You can use one of the following methods to resume the full and incremental synchronization task:

Forcefully rerun the full and incremental synchronization task

• Scenarios

  • The real-time synchronization subtask generated by the synchronization task fails for a long period of time. As a result, the binary logs are deleted, and incremental data in the source cannot be synchronized.

  • Destination tables do not contain the fields that are newly added to source tables due to various reasons.

  • Data accuracy issues such as data loss occur on data synchronized to destination tables due to various reasons.

• Operation

  1. Go to the Nodes page of Data Integration in the DataWorks console and find the full and incremental synchronization task.

  2. Click More in the Actions column of the task and select Force Rerun to forcefully rerun the task to re-synchronize full data and incremental data from all source tables to destination tables.

     You can find the full and incremental synchronization task on the Nodes page and click Running Details in the Actions column to view the running details of the task.

     Important

     • Before you forcefully rerun a full and incremental synchronization task used to synchronize data to MaxCompute, you must check whether the rerun operation will lead to a conflict between the instances of the merge subtasks generated by the task before and during the forceful rerun. When you forcefully rerun the task, the instance of the merge subtask generated before the forceful rerun may be running or be going to run. If the data timestamps of the merge subtasks that are generated by the task before and during the forceful rerun are the same and the merge subtasks are run at the same time, data in destination partitions or tables may overwrite each other.

       You can go to the View auto triggered node instances page in Operation Center and view the running situation of the instance of the merge subtask that is generated by the full and incremental synchronization task before the forceful rerun. If the rerun operation will lead to a conflict between the instances of the merge subtasks generated by the task before and during the forceful rerun, you can perform one of the following operations to resolve the issue:

       • If the instance of the merge subtask that is generated by the full and incremental synchronization task before the forceful rerun is running, rerun the task after the instance finishes running.

       • If the instance of the merge subtask that is generated by the full and incremental synchronization task before the forceful rerun has not started to run, freeze the instance. Unfreeze the instance after the rerun operation is complete.

     • A synchronization task that is used to synchronize data from tables in sharded databases cannot be forcefully rerun.

     • After you trigger a forceful rerun for a synchronization task, we recommend that you do not perform operations on the task, such as adding tables to or removing tables from the task. If you want to perform operations on the task, wait until the forceful rerun is complete. If you perform other operations on the task during a forceful rerun, the forceful rerun is invalid. You must trigger another forceful rerun for the task.

     • If data is not generated or the automatic scheduling of the merge subtask is not resumed on the next day after you forcefully rerun a synchronization task, you must check whether the following issues exist and manually resume the scheduling of the instance of the merge subtask:

       • If latency occurs on the synchronization task, resolve the latency issue. For more information, see Solutions to latency on a real-time synchronization node.

       • If the instance of the merge subtask in the previous cycle is not run or failed to be run, you can remove the dependency of the instance of the merge subtask in the current cycle on the instance of the merge subtask in the previous cycle. For more information, see View auto triggered node instances.

Backfill full data for the full and incremental synchronization task

1. Restart the real-time synchronization subtask generated by the task.

   Go to the Real Time DI page of Operation Center in the DataWorks console, find the real-time synchronization subtask generated by the task, and then click Start in the Actions column to restart the subtask and resume the synchronization of incremental data.

2. Backfill full data for the full and incremental synchronization task.

   1. Go to the Nodes page of Data Integration in the DataWorks console and find the full and incremental synchronization task.

   2. Click More in the Actions column and select Backfill Full Data. In the Backfill Full Data dialog box, configure the settings for full data backfill.

      1. Select the data timestamp of the data backfill instance.

         Select the date of the previous day as the data timestamp.

      2. Select source tables based on which you want to backfill full data.

         In the list on the left, select the tables from which you want to synchronize full data. If you want to recover the entire full and incremental synchronization task, select all tables. Click the icon to move the selected tables to the list on the right.

      3. Click OK.

   3. View the execution details.

      You can find the full and incremental synchronization task on the Nodes page and click Running Details in the Actions column to view the running details of the task.

      Important

      • Before you backfill full data for a full and incremental synchronization task used to synchronize data to MaxCompute, you must check the data timestamp of the data backfill instance. You must make sure that the data backfill instance does not conflict with the instance of the merge subtask generated before the full data backfill operation. When you backfill full data for the synchronization task, the instance of the merge subtask generated before the full data backfill may be running or be going to run. If the data timestamps of the instances are the same and the instances are run at the same time, data in destination partitions or tables may overwrite each other.

        You can go to the View auto triggered node instances page in Operation Center and view the running situation of the instance of the merge subtask. If the data backfill instance conflicts with the instance of the merge subtask, you can perform one of the following operations to resolve the issue:

        • If the instance of the merge subtask is running, backfill full data for the task after the instance finishes running.

        • If the instance of the merge subtask has not started to run, freeze the instance. Unfreeze the instance after the full data backfill operation is complete.

      • Synchronization tasks that are used to synchronize data from tables in sharded databases do not support full data backfill.

      • Before you perform full data backfill, you must make sure that you are familiar with the related precautions. In addition, you must check the data backfill result on the next day at the earliest opportunity. If data is not generated or the automatic scheduling of the merge subtask is not resumed on the next day after the data backfill, you must manually resume the scheduling of the instance of the merge subtask.

Create another full and incremental synchronization task

1. Go to the Real Time DI page of Operation Center in the DataWorks console, find the real-time synchronization subtask generated by the full and incremental synchronization task, and then click Stop in the Actions column to stop the running of the subtask.

2. Go to the DataStudio page and find the offline subtasks that are generated by the full and incremental synchronization task, including the check done subtasks and the merge subtask. Delete the subtasks and deploy the deletion operations.

   You can find the offline subtasks in the DataStudio workflow that is specified or automatically created for the subtasks.

3. Create another full and incremental synchronization task for which the generation method of destination tables is set to Use Existing Table, and run the task.

Recover data synchronization for some tables

1. Go to the Nodes page of Data Integration in the DataWorks console, find the full and incremental synchronization task, click More in the Actions column, and then select Edit. On the configuration page of the task, remove the source tables whose data fails to be synchronized from the task, save the modification, and then submit the task for running. Go to the configuration page in the same way, add the removed source tables to the task, save the modification, and then submit the task for running. This way, data synchronization from the tables can be recovered.

2. Recover synchronization of data to a historical partition.

   • For a full and incremental synchronization task used to synchronize all data from a database to MaxCompute, you can refer to the Backfill full data for the full and incremental synchronization task subsection in this topic to backfill full data for the task to synchronize full data in the source to a historical partition in the destination. You cannot backfill historical data based on a time range.

   • For other types of synchronization tasks, you must create a batch synchronization task to backfill historical data.

Scenarios in which an error occurs on the merge subtask

For information about how to resolve the issue, see Troubleshoot issues of a merge subtask generated by a full and incremental synchronization task used to synchronize data to MaxCompute.