Tunnel Service is a centralized service that uses the Tablestore API to allow you to consume full and incremental data. Tunnel Service provides tunnels that are used to export and consume full, incremental, and differential data. After you create a tunnel for a data table, you can use the tunnel to consume full and incremental data in the data table.
Background information
Tablestore is applicable to scenarios such as metadata management, time series data monitoring, and message systems. In these scenarios, incremental or differential data streams are often used to trigger the following operations:
Data synchronization: synchronizes data to a cache, search engine, or data warehouse.
Event triggering: triggers Function Compute, sends a notification when data is consumed, or calls an API operation.
Stream data processing: connects to a stream-computing engine or a unified batch and stream computing engine.
Data migration: backs up data to Object Storage Service (OSS) or migrates data to a Tablestore capacity instance.
Features
Tunnel Service provides tunnels for full and incremental data consumption, orderly incremental data consumption, consumption latency monitoring, and horizontal scaling of data consumption capabilities. The following table describes the features.
In scenarios in which 100,000 rows are written to a table per second, Tunnel Service provides a latency of milliseconds from when the data is updated to when the update record is obtained. The update record is returned in the sequence in which the data is updated.
Feature | Description |
Tunnels for full and incremental data consumption | Tunnel Service supports incremental data consumption and allows you to concurrently consume full data and differential data. |
Orderly incremental data consumption | Tunnel Service sequentially distributes incremental data to one or more logical partitions based on the write time. Data in different logical partitions can be concurrently consumed. |
Consumption latency monitoring | Tunnel Service allows you to call the DescribeTunnel operation to view the latency of the consumed data on each client. Tunnel Service also allows you to monitor data that is consumed by using tunnels in the Tablestore console. |
Horizontal scaling of data consumption capabilities | Tunnel Service supports automatic load balancing among logical partitions. This allows you to add more tunnel clients to accelerate data consumption. |
Usage notes
The incremental logs for a tunnel are retained for a maximum of seven days. The specific storage duration of incremental logs is consistent with that of the stream logs for a data table. If you create a tunnel to consume differential or incremental data, take note of the following items:
During full data consumption, if the tunnel does not complete the consumption of full data within seven days, an OTSTunnelExpired error occurs when the tunnel starts to consume incremental data. As a result, the tunnel cannot consume incremental data. If you estimate that full data cannot be consumed within seven days, contact Tablestore technical support.
During incremental data consumption, if the tunnel has not consumed incremental data for more than seven days, the consumption starts from the latest data that can be consumed. In this case, some data may not be consumed.
NoteIf incremental data has not been consumed for more than seven days, the incremental data is expired. The specific storage duration of incremental logs is consistent with that of the stream logs for a data table. If the incremental data is expired for the specified period, which is seven days by default, Tablestore disables the tunnel. In this case, the tunnel cannot be used to consume data.
Use Tunnel Service
You can use Tunnel Service by using the Tablestore console, Tablestore CLI, or Tablestore SDKs.
Billing
You are not charged for Tunnel Service. However, you are charged for the read throughput that is generated when you use tunnels to consume data. For more information, see Billing overview.