Tablestore:Use the Tablestore console to deliver data to OSS

Prerequisites

OSS is activated. A bucket is created in the region where a Tablestore instance resides. For more information, see Activate OSS.

Usage notes

• Data delivery is available in the China (Hangzhou), China (Shanghai), China (Beijing), and China (Zhangjiakou) regions.

• The delete operation on Tablestore data is ignored when the data is delivered. Tablestore data delivered to OSS is not deleted when you perform a delete operation on the data.

• It takes at most one minute for initialization when you create a delivery task.

• Latencies are within 3 minutes when data is written at a steady rate. The P99 latency is within 10 minutes when data is synchronized.

  Note

  The P99 latency indicates the average latency of the slowest 1% of requests over the previous 10 seconds.

Create a data delivery task

1. Go to the Instance Management page.

   1. Log on to the Tablestore console.

   2. In the upper part of the page, select a resource group and a region. Find a Tablestore instance that you want to manage in the instance list. Then, click the instance name or Manage Instance in the Actions column.

2. On the Instance Management page, click the Deliver Data to OSS tab.

3. (Optional) Create the service-linked role AliyunServiceRoleForOTSDataDelivery.

   When you configure data delivery for the first time, you must create the AliyunServiceRoleForOTSDataDelivery role that is used to authorize Tablestore to write data to an OSS bucket. For more information, see AliyunServiceRoleForOTSDataDelivery role.

   Note

   For more information about service-linked roles, see Service-linked roles.

   1. On the Deliver Data to OSS tab, click Role for Delivery Service.

   2. In the Role Details message, view related information. Click OK.

4. Create a delivery task.

   1. On the Deliver Data to OSS tab, click Create Task.

   2. In the Create Task dialog box, configure the following parameters.

      Parameter	Description
      Task Name	The name of the delivery task. The name must be 3 to 16 characters in length and can contain only lowercase letters, digits, and hyphens (-). It must start and end with a lowercase letter or digit.
      Destination Region	The region where the Tablestore instance and OSS bucket are located.
      Source Table	The name of the Tablestore table.
      Destination Bucket	The name of the OSS bucket to which you want to deliver data. Important Make sure that a bucket is created in the same region of the Tablestore instance.
      Destination Prefix	The prefix of the folder in the bucket. Data is delivered from Tablestore to the folder. The path of the destination folder supports the following time variables: $yyyy, $MM, $dd, $HH, and $mm. For more information, see the Partition data by time section of this topic. When the path uses time variables, OSS folders are dynamically generated based on the time at which data is written. This way, the data in OSS are organized, partitioned, and distributed based on time, which follows the hive partition naming style. When the path does not use time variables, all files are delivered to an OSS folder whose name contains this prefix.
      Synchronization Mode	The type of the delivery task. Valid values: Incremental: Only incremental data is synchronized. Full: All data in tables is scanned and synchronized. Differential: Full data is synchronized before incremental data is synchronized. When Tablestore synchronizes incremental data, you can view the time when data is last delivered and the status of the delivery task.
      Destination Object Format	The delivered data is stored in the Parquet format. By default, data delivery uses PLAIN for encoding. PLAIN can be used to encode data of all types.
      Schema Generation Type	Specify the columns to be delivered. The order in which fields are sorted in the Tablestore table can be different from the order of fields in the schema. Parquet data stored in OSS is distributed based on the order of fields in the schema. Select a schema generation type. If the Schema Generation Type parameter is set to Manual, you must configure the source fields, destination field names, and destination field types for delivery. If the Schema Generation Type parameter is set to Auto Generate, the system identifies and matches the fields for delivery. Important The data types must be consistent between the source and destination fields. Otherwise, the fields are discarded as dirty data. For more information about field type mappings, see the Data type mapping section of this topic. When you configure the schema, you can perform the following operations: Click +Add Field to add fields for delivery. Click the or icon in the Actions column corresponding to a field to adjust the order of the field. Click the icon in the Actions column corresponding to a field to delete the field.
      Schema Configurations

5. Click OK.

   In the View Statement to Create Table message, you can view the statement that is used to create an external table for E-MapReduce (EMR). You can also copy the statement to create an external table for EMR to access data in OSS.

   After the delivery task is created, you can perform the following operations:

   • View the details of the delivery task, such as the task name, table name, destination bucket, destination prefix, status, and the time when data is last synchronized.

   • View or copy the statement to create a table.

     Click View Statement to Create Table in the Actions column. You can view or copy the statement to create an external table by using computing engines such as EMR. For more information, see Use EMR.

   • View the error message returned after the delivery.

     If the configurations for the OSS bucket and delivery permissions are incorrect, data delivery cannot be complete. On the status page of the delivery task, you can view related error messages. For more information about error handling, see the Error handling section of this topic.

   • Delete the delivery task.

     Click Delete in the Actions column corresponding to the delivery task. You can delete the delivery task. The system returns an error if the delivery task is in the initialization process. You can delete the task later.

View OSS data

After the delivery task is initialized and data is delivered, you can view the data delivered to OSS by using the OSS console, API, SDK, or computing engine EMR. For more information, see Overview.

Example of an OSS object URL:

oss://BucketName/TaskPrefix/TaskName_ConcurrentID_TaskPrefix__SequenceID

In the example, BucketName indicates the name of the bucket. The first TaskPrefix indicates the prefix of the destination folder. The second TaskPrefix indicates the prefix information of the task. TaskName indicates the name of the task. ConcurrentID indicates the number for concurrency determined by the system. The number starts from 0 and increases when the throughput increases. SequenceID indicates the sequence ID of the delivered file and increases from 1.

Partition data by time

Data delivery allows the system to record the time when data is written to Tablestore. The time consists of the following variables: $yyyy (four-digit year), $MM (two-digit month), $dd (two-digit day), $HH (two-digit hour), and $mm (two-digit minute). The time can be used as the prefix of the destination bucket after conversion.

The delivery task by which data was written to Tablestore at 16:03 on August 31, 2020 is used in the example. The following table describes the URLs of the first object generated in OSS based on different destination prefix configurations.

Data type mapping

Error handling