DataWorks provides Hologres Reader and Hologres Writer for you to read data from and write data to Hologres data sources. This topic describes the capabilities of synchronizing data from or to Hologres data sources.
Supported Hologres versions
Hologres V0.7, V0.8, V0.9, V0.10, V1.1, V1.2, and V1.3
Limits
Batch data read and write
Hologres data sources support only exclusive resource groups for Data Integration.
Hologres Writer cannot write data to Hologres foreign tables.
Logic based on which the endpoint of a Hologres data source is obtained during the establishment of a network connection between the Hologres data source and a resource group:
If the Hologres data source resides in the same region as the resource group, the endpoint of the Hologres data source is obtained in the following order: .
If the Hologres data source resides in a different region from the resource group, the endpoint of the Hologres data source is obtained in the following order: .
Real-time data write in a database
Supported data types
Data type | Hologres Reader for batch data read | Hologres Writer for batch data write | Hologres Writer for real-time data write |
Data type | Hologres Reader for batch data read | Hologres Writer for batch data write | Hologres Writer for real-time data write |
UUID | Not supported | Not supported | Not supported |
CHAR | Supported | Supported | Supported |
NCHAR | Supported | Supported | Supported |
VARCHAR | Supported | Supported | Supported |
LONGVARCHAR | Supported | Supported | Supported |
NVARCHAR | Supported | Supported | Supported |
LONGNVARCHAR | Supported | Supported | Supported |
CLOB | Supported | Supported | Supported |
NCLOB | Supported | Supported | Supported |
SMALLINT | Supported | Supported | Supported |
TINYINT | Supported | Supported | Supported |
INTEGER | Supported | Supported | Supported |
BIGINT | Supported | Supported | Supported |
NUMERIC | Supported | Supported | Supported |
DECIMAL | Supported | Supported | Supported |
FLOAT | Supported | Supported | Supported |
REAL | Supported | Supported | Supported |
DOUBLE | Supported | Supported | Supported |
TIME | Supported | Supported | Supported |
DATE | Supported | Supported | Supported |
TIMESTAMP | Supported | Supported | Supported |
BINARY | Supported | Supported | Supported |
VARBINARY | Supported | Supported | Supported |
BLOB | Supported | Supported | Supported |
LONGVARBINARY | Supported | Supported | Supported |
BOOLEAN | Supported | Supported | Supported |
BIT | Supported | Supported | Supported |
JSON | Supported | Supported | Supported |
JSONB | Supported | Supported | Supported |
How Hologres Reader and Hologres Writer work
Batch data read and write
Hologres Reader reads data from Hologres tables by using PostgreSQL statements. The number of parallel threads that are used to read data is based on the number of shards in the Hologres table from which you want to read data. One SELECT statement is executed for each shard.
When you execute the CREATE TABLE
statement to create a table in Hologres, you can use the CALL set_table_property('table_name', 'shard_count', 'xx')
command to configure the number of shards for the table.
By default, the shard_count field is set to the default number of table shards for your Hologres database. The configurations of your Hologres instance determine the default number of table shards for your Hologres database.
A SELECT statement uses the shard that is specified by the built-in field hg_shard_id of the source Hologres table to query data.
Batch data write
Hologres Writer obtains data from a reader and writes data to Hologres based on the conflict processing mode that is configured for the related batch synchronization task.
You can configure a conflict processing mode for the batch synchronization task to control how to process conflicting data in a primary key conflict.
Important
You can configure a conflict processing mode only for a table that has a primary key. For information about the principle based on which data is written to Hologres when a primary key conflict occurs, and data write performance, see How it works.
If you configure the replacement mode for the batch synchronization task, new data overwrites the existing data, and all fields in a row are overwritten. NULL is written to the fields that have no mapped source fields as the value.
If you configure the update mode for the batch synchronization task, new data overwrites the existing data, and only fields that have mapped fields are overwritten.
If you configure the ignoring mode for the batch synchronization task, existing data is retained, and new data is ignored.
Develop a data synchronization task
For information about the entry point for and the procedure of configuring a data synchronization task, see the following sections. For information about the parameter settings, view the infotip of each parameter on the configuration tab of the task.
Add a data source
Before you configure a data synchronization task to synchronize data from or to a specific data source, you must add the data source to DataWorks. For more information, see Add and manage data sources.
Configure a batch synchronization task to synchronize data of a single table
Configure a real-time synchronization task to synchronize data of a single table or synchronize all data in a database
For more information about the configuration procedure, see Configure a real-time synchronization task in DataStudio.
Configure synchronization settings to implement real-time synchronization of full data or incremental data in a single table or a database
For more information about the configuration procedure, see Configure a synchronization task in Data Integration.
Appendix: Code and parameters
Appendix: Configure a batch synchronization task by using the code editor
If you use the code editor to configure a batch synchronization task, you must configure parameters for the reader and writer of the related data source based on the format requirements in the code editor. For more information about the format requirements, see Configure a batch synchronization task by using the code editor. The following information describes the configuration details of parameters for the reader and writer in the code editor.
Code for Hologres Reader
Parameters in code for Hologres Reader
Parameter | Description | Required | Default value |
Parameter | Description | Required | Default value |
endpoint | The endpoint of the source Hologres instance. Specify the value in the format of instance-id-region-endpoint.hologres.aliyuncs.com:port . You can obtain the endpoint of a Hologres instance on the details page of the instance in the Hologres console. The endpoint of a Hologres instance varies based on the network type, including the classic network, Internet, and VPC. Configure this parameter based on the type of the network where your exclusive resource group for Data Integration and the Hologres instance reside. If an invalid endpoint is specified, the connection between the exclusive resource group for Data Integration and the Hologres instance may fail, or data synchronization performance may be poor. The endpoints for the three network types are in the following formats: Classic network endpoint: instance-id-region-endpoint-internal.hologres.aliyuncs.com:port Public endpoint: instance-id-region-endpoint.hologres.aliyuncs.com:port VPC endpoint: instance-id-region-endpoint-vpc.hologres.aliyuncs.com:port
We recommend that you deploy the exclusive resource group for Data Integration and the Hologres instance in the same zone of the same region. This helps ensure a successful network connection and the optimal data synchronization performance. | Yes | No default value |
accessId | The AccessKey ID of the account that you use to connect to Hologres. | Yes | No default value |
accessKey | The AccessKey secret of the account that you use to connect to Hologres. Make sure that the account is authorized to read data from the source table. | Yes | No default value |
database | The name of the source database in the Hologres instance. | Yes | No default value |
table | The name of the table from which you want to read data. If the table is a partitioned table, specify the name of the table instead of the name of the partition from which you want to read data. | Yes | No default value |
column | The names of the columns from which you want to read data. The names of the primary key columns in the source table must be included. If you want to read data from all columns in the source table, set this parameter to ["*"] . | Yes | No default value |
partition | The partition key column and the related value of the source table, in the format of column=value . This parameter is valid only for partitioned tables. Important Hologres supports only list partitioning. You can specify only one column as the partition key column. The data type of the partition key column must be INT4 or TEXT. The value of this parameter must match the partition filter expression in the DDL statements that are used to create the destination Hologres table. You must specify a child partitioned table that exists and contains data.
| No | Left empty, which indicates that the source table is a non-partitioned table |
fetchSize | The maximum number of data records to read from the source table at a time by using the SELECT statement. | No | 1,000 |
Code for Hologres Writer
Parameters in code for Hologres Writer
Parameter | Description | Required | Default value |
Parameter | Description | Required | Default value |
endpoint | The endpoint of the destination Hologres instance. Specify the value in the format of instance-id-region-endpoint.hologres.aliyuncs.com:port . You can obtain the endpoint of a Hologres instance on the details page of the instance in the Hologres console. The endpoint of a Hologres instance varies based on the network type. Network types include the Internet, classic network, and VPC. Configure this parameter based on the type of the network in which your exclusive resource group for Data Integration and the Hologres instance reside. If an invalid endpoint is specified, the network connection between the exclusive resource group for Data Integration and the Hologres instance may fail, or data synchronization performance may be degraded. The endpoints for the three network types are in the following formats: Public endpoint: instance-id-region-endpoint.hologres.aliyuncs.com:port Classic network endpoint: instance-id-region-endpoint-internal.hologres.aliyuncs.com:port VPC endpoint: instance-id-region-endpoint-vpc.hologres.aliyuncs.com:port
We recommend that you deploy the exclusive resource group for Data Integration and the Hologres instance in the same zone of the same region. This helps ensure a successful network connection and the optimal data synchronization performance. | Yes | No default value |
accessId | The AccessKey ID of the account that you use to connect to Hologres. | Yes | No default value |
accessKey | The AccessKey secret of the account that you use to connect to Hologres. Make sure that the account is authorized to write data to the destination Hologres table. | Yes | No default value |
database | The name of the destination database in the Hologres instance. | Yes | No default value |
table | The name of the Hologres table to which you want to write data. You can specify the table name in the format of Schema_name.Table_name . | Yes | No default value |
conflictMode | The conflict processing mode. Valid values: Replace, Update, and Ignore. For more information, see How Hologres Reader and Hologres Writer work. | Yes | No default value |
column | The names of the columns to which you want to write data. The names of the primary key columns in the destination Hologres table must be included. If you want to write data to all the columns in the destination Hologres table, set this parameter to ["*"] . | Yes | No default value |
partition | The partition key column and the related value in the destination Hologres table. Configure this parameter in the format of column=value . This parameter is valid only for partitioned tables. Note Hologres supports only list partitioning. You can specify only one column as the partition key column. The data type of the partition key column must be INT4 or TEXT. The value of this parameter must match the partition filter expression in the DDL statements that are used to create the destination Hologres table.
| No | Left empty, which indicates that the destination Hologres table is a non-partitioned table |
truncate | Specifies whether to delete the existing data in the destination Hologres table before Hologres Writer writes data to the table. | No | false |