This topic describes how Lindorm Streams provided by ApsaraDB for Lindorm is used to publish messages to topics of Message Queue for Apache Kafka. The messages contain incremental data from ApsaraDB for HBase Performance-enhanced. This way, you can subscribe to the topics and consume the incremental data.
Features
Lindorm Streams publishes messages to topics of Message Queue for Apache Kafka based on the following rules:
Lindorm Streams records operations that are performed on the same row in the order in which these operations are performed. This way, the recorded operations can be re-performed in the same order.
The At-least-once semantics is supported. Messages can be consumed multiple times based on the retry policy that you configure. If you want to specify the exactly-once semantics, submit a ticket.
Message format
A Lindorm Streams message can contain two versions of an updated row: the version before the row is updated and the version after the row is updated. The following example shows the content of a message:
{
"op" : <opType> // The operation that is performed. Valid values: Put and DeleteFamily.
"table" : <tableName> // The name of the table.
"ts" : <defaultVersion> // The default version. The default version is used if the row does not contain the ts key.
"keyOnly":<keyOnly> // Indicates whether only the primary key is included.
"data" : [
{
"type" : <type> // The data type of the column.
"name" : <name> // The name of the column.
"ts" : <version> // The data version. If this parameter is empty, the dts value is used as the version.
"value" : <value> // The value of the column. If this parameter is empty, the actual value in the table is null.
},
{
"type" : <type> // The data type of the column.
"name" : <name> // The name of the column.
"ts" : <version> // The data version. If this parameter is empty, the dts value is used as the version.
"value" : <value> // The value of the column. If this parameter is empty, the actual value in the table is null.
}
]
}
The following table describes the parameters that you can specify for Lindorm Streams messages.
Parameter | Type | Required | Example | Description |
key_only | boolean | No | false | Specifies whether to synchronize only primary keys. Default value: false. |
unique_key | boolean | No | false | Specifies whether a primary key is sent only once per batch. Default value: false. |