Tablestore:Data Integration

Make sure that the following preparations are made before you perform subsequent operations:

• Information about the MaxCompute table that you want to synchronize to a Tablestore table is confirmed and recorded.

• A RAM user is created and granted Tablestore management permissions (AliyunOTSFullAccess) and DataWorks management permissions (AliyunDataWorksFullAccess). For more information, see Create a RAM user and Grant permissions to the RAM user.

• An AccessKey pair is created for the RAM user. For more information, see Create an AccessKey pair.

• DataWorks is activated and a workspace is created. For more information, see Activate DataWorks and Create a workspace.

• Tablestore is activated. An instance and a data table are created. For more information, see Step 1: Activate Tablestore, Step 2: Create an instance, and Step 3: Create a data table.

  Important

  The names and types of the primary key columns of the Tablestore data table must match the names and types of the primary key columns of the MaxCompute table. You do not need to define attribute columns in the Tablestore data table. You can specify the names of attribute columns when you write data to the Tablestore data table.

In this example, a Tablestore data source is added. Add the required data sources based on your business requirements.

1. Go to the Data Integration page.

   1. Log on to the DataWorks console as the project administrator.

   2. In the left-side navigation pane, click Workspaces. In the top navigation bar, select a region.

   3. On the Workspaces page, find the workspace that you want to manage and choose Shortcuts > Data Integration in the Actions column.

2. In the left-side navigation pane, click Data Source.

3. On the Data Source page, click Add Data Source.

4. In the Add Data Source dialog box, click the Tablestore block.

5. In the Add OTS data source dialog box, configure the parameters that are described in the following table.

   Parameter	Description
   Data Source Name	The name of the data source. The name can contain letters, digits, and underscores (_), and must start with a letter.
   Data Source Description	The description of the data source. The description cannot exceed 80 characters in length.
   Endpoint	The endpoint of the Tablestore instance. For more information, see Endpoints. If the Tablestore instance and the resources of the destination data source are in the same region, enter a virtual private cloud (VPC) endpoint. Otherwise, enter a public endpoint.
   Table Store instance name	The name of the Tablestore instance. For more information, see Instance.
   AccessKey ID	The AccessKey ID and AccessKey secret of your Alibaba Cloud account or RAM user. For more information about how to create an AccessKey pair, see Create an AccessKey pair.
   AccessKey Secret

6. Test the network connectivity between the data source and the resource group that you select.

   To ensure that your synchronization nodes run as expected, you need to test the connectivity between the data source and all types of resource groups on which your synchronization nodes will run.

   Important

   A synchronization task can use only one type of resource group. By default, only shared resource groups for Data Integration are displayed in the resource group list. To ensure the stability and performance of data synchronization, we recommend that you use an exclusive resource group for Data Integration.

   1. Click Purchase to create a new resource group or click Associate Purchased Resource Group to associate an existing resource group. For more information, see Create and use an exclusive resource group for Data Integration.

   2. Find the resource group that you want to manage and click Test Network Connectivity in the Connection Status column.

      If Connected is displayed in the Connection Status column, the connectivity test is passed.

7. If the data source passes the network connectivity test, click Complete.

   The newly created data source is displayed in the data source list.

1. Add a synchronization task node.

   1. Go to the DataStudio console.

      1. Log on to the DataWorks console as a project administrator.

         Important

         Only the project administrator role can be used to add data sources. Members who are assigned other roles can only view data sources.

      2. In the left-side navigation pane, click Workspaces. In the top navigation bar, select a region.

      3. On the Workspaces page, find the workspace that you want to manage and choose Shortcuts > Data Development in the Actions column.

   2. In the Scheduled Workflow section of the DataStudio console, click Business Flow and select a business flow.

      For information about how to create a workflow, see Create a workflow.

   3. Right-click Data Integration and choose Create Node > Offline synchronization.

   4. In the Create Node dialog box, enter a name in the Name field and click Confirm.

      You can find the offline synchronization node that you created under Data Integration.

2. Configure network and resource links.

   Select the source and destination for the batch synchronization task, and the resource group that is used to run the batch synchronization task. Establish network connections between the resource group and data sources, and test the connectivity.

   1. Double-click the created batch synchronization node under Data Integration.

   2. In the Configure Network Connections and Resource Group step, set the Source parameter to MaxCompute(ODPS) and the Data Source Name parameter to the name of the MaxCompute data source.

   3. Select a resource group.

      After you select a resource group, the system displays the region and specifications of the resource group and automatically tests the connectivity between the resource group and the selected data source.

      Important

      Make sure that the resource group is the same as the resource group that you selected when you added the data source.

   4. Set the Destination parameter to Tablestore and the Data Source Name parameter to the name of the Tablestore data source.

      The system automatically tests the connectivity between the resource group and the destination.

   5. After the connectivity test is passed, click Next.

3. Configure the source and destination.

   1. In the Configure Source and Destination section, configure the parameters for the source. The following table describes the parameters.

      Parameter	Description
      Tunnel Resource Group	The data transmission resources of MaxCompute. The default value is Common transmission resources, which is a free quota for MaxCompute.
      Table	The name of the table in MaxCompute.
      Partition information	The names of the partitions from which you want to read data.

   2. In the Configure Source and Destination section, configure the parameters for the destination. The following table describes the parameters.

      Parameter	Description
      Table	The name of the Tablestore data table.
      primaryKeyInfo	The primary key information about the Tablestore data table. The value of the parameter is a JSON array. The primary key of a data table consists of one to four primary key columns. Specify the primary key information based on the actual number of primary key columns. Important The number, sequence, and types of primary key columns that you specify must be the same as the number, sequence, and types of primary key columns in the data table. Example: [ { "name": "pk", "type": "string" } ]
      WriteMode	The mode in which data is written to Tablestore. Valid values: PutRow: inserts data into the specified row. This mode corresponds to the Tablestore PutRow API operation. If the specified row does not exist, a new row is added. If the specified row exists, the row is overwritten. UpdateRow: updates data in the specified row. This mode corresponds to the Tablestore UpdateRow API operation. If the specified row does not exist, a new row is added. If the specified row exists, the values of the specified columns in the row are added, modified, or removed based on the content of the request.

4. Configure field mappings.

   After you configure the source and destination, you must configure mappings between source fields and destination fields. After you configure the mappings, the batch synchronization task writes the values of the source fields to the destination fields of the specific data type based on the mappings. By default, the system maps the field in a row of the source to the field in the same row of the destination.

   Important

   • If you want to change the mapping sequence, you can click the icon next to Target field that corresponds to Source field. You can specify the name of a source field to represent the source field. You must use the JSON format to specify a destination field. Example: {"name":"age","type":"INTEGER"}. Each line represents a field.

   • The data types of attribute columns in the Tablestore data table must match the data types of fields that you want to write to MaxCompute. The names of attribute columns in the Tablestore data table can be different from the names of the fields that you want to write to MaxCompute.

   By default, the primary key information about the destination is specified when you configure the parameters for the destination. In this case, you must manually delete primary key information in the destination fields. Otherwise, the batch synchronization task fails.

   1. Click the icon next to Target field.

   2. In the Manually Edit Table Fields dialog box, delete the primary key information in the destination fields and add or modify the attribute column information based on your business requirements.

   3. Click Confirmation.

5. Configure channel control policies.

   In the Channel Control section, configure the parameters for task execution, such as the Synchronization rate and Policy for Dirty Data Records parameters. For more information, see Step 5: Configure channel control policies.

6. Click the icon to save the configurations.

   Note

   If you do not save the configurations, a message that prompts you to save the configurations appears when you perform subsequent operations. In this case, click Confirmation in the message to save the configurations.

7. Run the synchronization task.

   Important

   In most cases, you need to synchronize full data only once. You do not need to configure scheduling properties.

   1. Click .

   2. In the Parameters dialog box, select the name of the resource group from the drop-down list.

   3. Click Run.

      After the synchronization task is complete, click the URL of the run log on the Runtime Log tab to go to the details page of the run log. On the details page of the run log, check the value of Current task status.

      If the value of Current task status is FINISH, the task is complete.

Log on to the Tablestore console to view data.

1. Log on to the Tablestore console.

2. In the upper part of the Overview page, select a region.

3. Click the name of the Tablestore instance to which data is restored.

4. On the Instance Details tab, click the name of the restored table on the Tables tab.

5. On the Query Data tab, you can view the data imported to Tablestore.