Lindorm:Data subscription

Background information

The change tracking feature records the change of each row of data in a Lindorm table. You can view the change records on your client in real time. After you enable the change tracking feature for a Lindorm table, each operation that modifies the data in the table is recorded. When you insert, update, or delete data, Lindorm generates a stream record that is a key-value pair. In the key-value pair, the key is the primary key value of the row in which the data resides, and the value includes the details of the operation. The details include the data before the operation is performed, the data after the operation is performed, the timestamp when the operation is performed, and the type of the operation.

Enable change tracking

You can use one of the following modes to enable change tracking for a Lindorm table: Pull and Push.

• Pull mode: You can use an open source Kafka client to subscribe to data. The subscribed data is temporarily stored on the Lindorm server. You must specify a duration for which you want to retain the subscribed data. When the data is retained for a period longer than the specified duration, the data is deleted. For more information, see Create a Pull channel for data subscription.

• Push mode: Lindorm pushes the subscribed data to your Message Queue for Apache Kafka topic. The Lindorm server does not store the subscribed data. For more information, see Create a Push channel for data subscription.

Limits

• The change tracking feature of Lindorm supports only the At Least Once data processing semantics. Data can be consumed multiple times based on the retry policy that you configure for Lindorm Tunnel Service (LTS).

• The order in which the incremental messages in the change tracking feature are consumed is different from the order in which the messages are written to the table.Lindorm

• The change tracking feature for Lindorm HBase database tables requires LTS 3.7.0 and later versions.

  Note

  • If you activated LTS, upgrade LTS to the latest version before you activate the change tracking feature. For more information, see Upgrade the minor engine version of a Lindorm instance.

  • If you did not activate LTS, you can purchase LTS when you enable the change tracking feature. By default, the latest version of LTS is provided. For more information, see Enable change tracking.

• If your Lindorm instance does not use SSDs, we recommend that you do not enable the change tracking feature. If you enable this feature for a table in a Lindorm instance that does not use SSDs, the database performance may be degraded. After you enable the change tracking feature for a table, each time the latest data in a row or data in a row before update is exported, Lindorm reads the data in the row before the data is exported. A read operation is performed each time when you perform a write operation. If you want to subscribe to only the delta data and do not want to use the OldImage/NewImage feature, submit a ticket.

• If you use the Pull mode to create data subscription channels, data is stored in LindormDFS for seven days by default. Make sure that your LindormDFS instance has sufficient storage space before you enable the change tracking feature.