Configure a batch synchronization task to synchronize data from tables in sharded databases

Go to the Data Integration page.

In the left-side navigation pane of the Data Integration page, choose Data Sources > Data Sources. On the Data Sources page, click Add Data Source. In the Add Data Source dialog box, select the desired data source type. This method supports only MySQL and PolarDB data sources.

Add each desired database to DataWorks as a common data source. In this example, MySQL sharded databases are added as common data sources. For more information, see Add and manage data sources in Data Integration.

Merge multiple common data sources into one sharded data source and use the sharded data source as the source of a synchronization task.

1. In the Add Data Source dialog box, click Sub-warehouse and sub-table.

2. Select a data source type based on your business requirements and configure the related parameters. In this example, MySQL is selected.

   

   Take note of the following parameters:

   • Data Source Name: Specify a name based on your business requirements.

   • Select Data Source: Select the common data sources that are added.

   • Meta Data Source: Select the data source that you want to use as the default metadata template. If you want to synchronize data from a sharded data source, make sure that the schemas of all tables in the selected common data sources are the same. After you specify a meta data source, the system can extract the default schema when you configure a synchronization task.

     Important

     If the schemas of tables in the data sources are different, an error is reported when the synchronization task is run.

Create a batch synchronization node.

Configure a synchronization task to synchronize data from tables in sharded databases.

Configure the synchronization task by using the codeless UI. In the Configure Network Connections and Resource Group step, select MySQL(Sharding) from the Source drop-down list, and then select the sharded data source that you added from the Data Source Name drop-down list. For more information about how to configure the synchronization task, see Configure a batch synchronization task by using the codeless UI.

Click Next.

In the Configure tasks step, select the tables from which you want to synchronize data and the table to which you want to write data. Then, save and deploy the synchronization task and perform subsequent operations.

Add each desired database to DataWorks as a common data source. In this example, MySQL sharded databases are added as common data sources. For more information, see Add and manage data sources in Data Integration.

Create a batch synchronization node.

Configure a synchronization task to synchronize data from tables in sharded databases.

Configure the synchronization task by using the codeless UI. In the Configure Network Connections and Resource Group step, click Edit Source to select multiple data sources after you select the source type. For more information about how to configure the synchronization task, see Configure a batch synchronization task by using the codeless UI.

In the Edit Data Source dialog box, select the desired data sources in the Data Sources to Be Selected list and click the rightward arrow to move the selected data sources to the Selected Data Sources list. Then, click OK.

Click Next.

In the Configure tasks step, select the tables from which you want to synchronize data in each selected data source in the Source section, and configure the parameters in the Destination section. Then, save and deploy the synchronization task and perform subsequent operations.

Create a batch synchronization node.

Configure a synchronization task to synchronize data from tables in sharded databases.

Configure the synchronization task by using the code editor. The following code provides a sample script. For more information about how to configure the synchronization task, see Configure a batch synchronization task by using the code editor.

{
    "type":"job",
    "version":"2.0",
    "steps":[
        {
            "stepType":"mysql",
            "parameter":{
                "envType":0,
                "column":[
                    "id",
                    "name"
                ],
                "socketTimeout":3600000,
                "tableComment":"",
                "connection":[    // The sharded data source.
                    {
                        "datasource": "datasourceName1"  // The name of the first sharded database.
                        "table":[           // The names of the source tables in the first sharded database.
                            "tb1"
                        ]
                    },
                    {
                        "datasource": "datasourceName2" // The name of the second sharded database.
                        "table":[          // The names of the source tables in the second sharded database.
                            "tb2",
                            "tb3"
                        ]
                    }
                ],
                "useSpecialSecret":true,// Specify whether to use separate passwords for the sharded databases.
                "where":"",
                "splitPk":"id",
                "encoding":"UTF-8"
                },
            "name":"Reader",
            "category":"reader"
            },
        {
            "stepType":"odps",
            "parameter":{
                "partition":"pt=${bizdate}",
                "truncate":true,
                "datasource":"odpsname",
                "envType":0,
                "isSupportThreeModel":false,
                "column":[
                    "id",
                    "name"
                ],
                "emptyAsNull":false,
                "tableComment":"",
                "table":"t1",
                "consistencyCommit":false
                },
            "name":"Writer",
            "category":"writer"
            }
    ],
    "setting":{
        "executeMode":null,
        "errorLimit":{
            "record":""
            },
        "speed":{
            "concurrent":2,
            "throttle":false
            }
    },
    "order":{
        "hops":[
            {
                "from":"Reader",
                "to":"Writer"
                }
        ]
    }
}