Tunnel Service is a centralized service that uses the Tablestore API to allow you to consume full and incremental data. Tunnel Service provides tunnels that are used to export and consume full, incremental, and differential data. After you create a tunnel for a data table, you can use the tunnel to consume full and incremental data in the data table.
Background information
Tablestore is applicable to scenarios such as metadata management, time series data monitoring, and message systems. In these scenarios, incremental or differential data streams are often used to trigger the following operations:
Data synchronization: synchronizes data to a cache, search engine, or data warehouse.
Event triggering: triggers Function Compute, notifies the consumer of consuming data, or calls an API operation.
Stream data processing: connects to a stream-computing engine or a unified batch and stream computing engine.
Data migration: backs up data to OSS or migrates data to a Tablestore capacity instance.
Features
Tunnel Service provides tunnels for full and incremental data consumption, orderly incremental data consumption, consumption latency monitoring, and horizontal scaling of data consumption capabilities. The following table describes the features.
In scenarios in which 100,000 rows are written to a table per second, Tunnel Service provides a latency of milliseconds from when the data is updated to when the update record is obtained. The update record is returned in the sequence in which the data is updated.
Feature | Description |
Tunnels for full and incremental data consumption | Tunnel Service supports incremental data consumption and allows you to concurrently consume full data and differential data. |
Orderly incremental data consumption | Tunnel Service sequentially distributes incremental data to one or more logical partitions based on the write time. Data in different logical partitions can be concurrently consumed. |
Consumption latency monitoring | Tunnel Service allows you to call the DescribeTunnel operation to view the latency of the consumed data on each client. Tunnel Service also allows you to monitor data that is consumed through tunnels in the Tablestore console. |
Horizontal scaling of data consumption capabilities | Tunnel Service supports automatic load balancing among logical partitions. This allows you to add more tunnel clients to accelerate data consumption. |
Usage notes
The retention period of incremental logs for tunnels is the same as the retention period of Stream logs. Stream logs can be retained for up to seven days. Therefore, incremental logs for tunnels can be retained for up to seven days.
If you create a tunnel to consume differential or incremental data, take note of the following items:
During full data consumption, if the tunnel does not complete the consumption of full data within the retention period of incremental logs (seven days at most), an
OTSTunnelExpirederror occurs when the tunnel starts to consume incremental logs. As a result, the tunnel cannot consume incremental logs.If you estimate that the tunnel cannot complete the consumption of full data within the specified time window, contact Tablestore technical support.
During incremental data consumption, if the tunnel does not complete the consumption of incremental logs within the retention period of incremental logs (up to seven days), the tunnel may consume data from the latest data that can be consumed. In this case, some data may not be consumed.
After a tunnel expires, Tablestore may disable the tunnel. If a tunnel remains in the disabled state for more than 30 days, the tunnel is deleted. You cannot restore a deleted tunnel.
Use Tunnel Service
You can use Tunnel Service in the Tablestore console or by using the Tablestore CLI or Tablestore SDKs.
Billing
You are not charged for Tunnel Service. However, you are charged for the read throughput that is generated when you use tunnels to consume data. For more information, see Billing overview.