Synchronize the real-time incremental data
The preceding figure shows that the incremental data consists of two portions: the data updates in the data source and the data pushed by using API operations. The incremental data is synchronized to OpenSearch in three steps:
1. The data updates in the data source are synchronized or the data is pushed by using API operations to an offline application in OpenSearch. You can subscribe to the binary logs of the data source in Data Transmission Service (DTS) to synchronize the data updates. The total number of transactions per second (TPS) in the primary and secondary tables cannot exceed 1,500.
2. When the incremental data is synchronized to the offline application, the incremental data is updated to an existing wide table. An update in a secondary table triggers N updates in the primary table. If the updates triggered in the primary table are greater than or equal to 1,000 TPS, the speed of updating secondary tables is limited to reduce the data synchronization latency in the primary table. For more information, see Data synchronization latency caused by multi-table joins.
3. The offline application writes the data that contains metadata to the engine. After the metadata is added, the data amount may be two or three times that in the data source. To protect the engine, the speed of writing data is limited to 10 MB/s.
Metrics
Metric | Maximum value | Remarks |
Total number of TPS in the primary and secondary tables when data is synchronized from the data source to the offline application. Unit: TPS. In this case, no trigger relationship is configured between the primary and secondary tables. | 1,500 | N/A |
Speed of writing the real-time incremental data from the offline application to the engine. Unit: MB/s. | 10 | Metadata is added. |
Updates in the primary table that are triggered by the updates in secondary tables. Unit: TPS. | 1,000 | N/A |