Hologres catalogs provide metadata, such as databases, tables, and partitions. The metadata also includes functions and information that are required to access data stored in a database or other external systems. After you create a Hologres catalog, you can directly read Hologres metadata in the development console of Realtime Compute for Apache Flink without the need to manually register Hologres tables. Hologres catalogs improve the efficiency of draft development and ensure data accuracy. This topic describes how to create, view, use, and drop a Hologres catalog in the development console of Realtime Compute for Apache Flink.
Limits
Only Flink 1.13 and later support the configuration of Hologres catalogs.
You cannot modify catalogs.
Only Realtime Compute for Apache Flink whose engine version is vvr-6.0.1-flink-1.15 or later allows you to configure table attributes when you create a table.
Only Realtime Compute for Apache Flink whose engine version is vvr-6.0.3-flink-1.15 or later allows you to configure the default parameters and table attributes of a Hologres table when you create a Hologres catalog.
Realtime Compute for Apache Flink allows you to read data from and write data to only Hologres internal tables. Therefore, only Hologres exclusive instances can be used when you create a Hologres catalog. Hologres Shared Cluster instances are not supported.
Create a Hologres catalog
You can create a Hologres catalog on the UI or by executing an SQL statement. We recommend that you create a Hologres catalog on the UI.
Create a Hologres catalog on the UI (recommended)
Go to the Catalogs page.
Log on to the management console of Realtime Compute for Apache Flink. Find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane, click Catalogs.
On the Catalog List page, click Create Catalog. In the Create Catalog dialog box, click Hologres on the Built-in Catalog tab in the Choose Catalog Type step and click Next.
Configure the parameters in the Configure Catalog step.
ImportantAfter you create a Hologres catalog, the parameter configuration cannot be modified. If you want to modify the parameter configuration, you must delete the Hologres catalog that you created and create a Hologres catalog again.
Parameter
Description
Required
Remarks
catalogname
The name of the Hologres catalog.
Yes
The name can contain only lowercase letters and digits. It cannot contain uppercase letters or special characters such as hyphens (-) and underscores (_).
endpoint
The endpoint of the Hologres instance.
Yes
For more information, see Instance configurations.
username
The AccessKey ID of the Alibaba Cloud account.
Yes
For more information, see Console operations.
NoteTo protect your AccessKey pair, we recommend that you configure the AccessKey secret by using variables. For more information, see Manage variables.
The Alibaba Cloud account to which the AccessKey ID belongs must be able to access all Hologres databases. For more information about Hologres database permissions, see Hologres permission models.
password
The AccessKey secret of the Alibaba Cloud account.
Yes
dbname
The name of the default Hologres database that you want to access.
Yes
N/A.
Click Confirm.
NoteYou can view the catalog that you create in the Catalogs pane on the left side of the Catalog List page.
Create a Hologres catalog by executing an SQL statement
In the code editor of the Scripts tab on the SQL Editor page, enter the following statement to create a Hologres catalog:
CREATE CATALOG <catalogname> WITH ( 'type' = 'hologres', 'endpoint' = '<endpoint>', 'username' = '<AccessKey>', 'password' = '<AccessSecret>', 'dbname' = '<dbname>' );
The following table describes the parameters.
Parameter
Description
Required
Remarks
catalogname
The name of the Hologres catalog.
Yes
The name can contain only lowercase letters and digits. It cannot contain uppercase letters or special characters such as hyphens (-) and underscores (_).
type
The type of the catalog.
Yes
Set the value to hologres.
endpoint
The endpoint of the Hologres instance.
Yes
For more information, see Instance configurations.
username
The AccessKey ID of the Alibaba Cloud account.
Yes
For more information, see Console operations.
NoteTo protect your AccessKey pair, we recommend that you configure the AccessKey secret by using variables. For more information, see Manage variables.
The Alibaba Cloud account to which the AccessKey ID belongs must be able to access all Hologres databases. For more information about Hologres database permissions, see Hologres permission models.
password
The AccessKey secret of the Alibaba Cloud account.
Yes
dbname
The name of the default Hologres database that you want to access.
Yes
N/A.
ignore-non-persisted-options
Specifies whether to ignore non-persistence options when you use a Hologres catalog to create a table that uses non-persistence options.
No
Valid values:
true: The table can be created and all non-persistence options are ignored. This is the default value.
false: An error that indicates that the table fails to be created is returned.
NoteThe persistence of Hologres catalog table options allows the system to reacquire the consistent information that is specified in the DDL statement when the system reads the table information from the catalog again. Only the following persistence options are supported: endpoint, username, password, and dbname.
Other parameters supported by the Hologres connector
When you create a Hologres catalog, you can configure the parameters, such as the parameters in the WITH clause of a DDL statement that is used to create a Hologres table. If you configure the parameters for the catalog, the parameters are automatically configured for the table in the Hologres catalog.
No
Only Realtime Compute for Apache Flink whose engine version is vvr-6.0.3-flink-1.15 or later supports this parameter.
NoteIf you want to configure the parameters, you must set the
ignore-non-persisted-options
parameter to true.In Realtime Compute for Apache Flink whose engine version is earlier than vvr-6.0.3-flink-1.15, you can only use SQL hints to configure the parameters for each table.
Select the code that is used to create a catalog and click Run that appears on the left side of the code.
View a Hologres catalog
After you create a Hologres catalog, you can perform the following operations to view the metadata in the Hologres catalog.
Go to the Catalogs page.
Log on to the management console of Realtime Compute for Apache Flink.
Find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane, click Catalogs.
On the Catalog List page, find the desired catalog and view the Catalog Name and Type columns of the catalog.
NoteIf you want to view the databases and tables in the catalog, click View in the Actions column.
If the public schema is used, the schema prefix is not added to a table name. In this case, only the table name is used.
Use a Hologres catalog
If the public schema is used, ${table_name} is used instead of ${schema_name.table_name}. This is because the prefix of the schema name is not added to the table name.
The update data in the table that is read by using a Hologres catalog can be consumed. By default, the ignoredelete property of the table is set to false and the mutatetype property of the table is set to insertorupdate. For more information about the ignoredelete and the mutatetype properties, see Merge data into a wide table.
Create a Hologres table
Create a Hologres table on the UI
Go to the Catalogs page.
Log on to the management console of Realtime Compute for Apache Flink.
Find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane, click Catalogs.
On the Catalog List page, find the desired catalog and click View in the Actions column.
On the page that appears, find the desired database and click View in the Actions column.
On the page that appears, click Create Table.
On the Built-in tab of the Create Table dialog box, click Connection Type and select a table type.
Click Next.
Enter the table creation statement and configure related parameters. Sample code:
CREATE TABLE `${catalog_name}`.`${db_name}`.`${table_name}` ( id INT, name STRING ) WITH ( 'connector' = 'hologres' );
Click Confirm.
Create a Hologres table by executing an SQL statement
In the code editor of the Scripts tab on the SQL Editor page, enter the following statement:
You can use one of the following methods to create a Hologres table:
Execute the
USE CATALOG HoloName
statement to directly reference the information of the Hologres service.USE CATALOG ${catalog_name}; CREATE TABLE `${db_name}`.`${schema_name.table_name}`( ... ) WITH ( 'connector' = 'hologres' );
For more information about the USE syntax, see USE statements.
Directly reference the information about the Hologres service in DDL statements.
CREATE TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` ( ... ) WITH ( 'connector' = 'hologres' );
Configure physical table properties in DDL statements.
CREATE TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` ( ... ) WITH ( 'connector' = 'hologres', 'table_property.orientation' = 'column', 'table_property.distribution_key' = 'a', 'table_property.clustering_key' = 'b:desc', 'table_property.bitmap_columns' = 'a,b', 'table_property.segment_key' = 'c', 'table_property.time_to_live_in_seconds' = '86400', 'table_property.binlog.level' = 'replica', 'table_property.binlog.ttl' = '86400' );
NoteWhen you create a table for a registered Hologres service, you must set the connector parameter to hologres in the WITH clause. Other parameters such as endpoint are optional.
When you create a Hologres table, specific physical table attributes of the table that you specify in the WITH clause cannot be modified.
Realtime Compute for Apache Flink whose engine version is earlier than vvr-6.0.3-flink-1.15 does not allow you to modify the physical table properties that are specified in the WITH clause when you create a physical table. You can log on to the Hologres console to modify the physical table properties. Only Realtime Compute for Apache Flink whose engine version is vvr-6.0.3-flink-1.15 or later allows you to modify physical table properties.
You cannot add or modify parameters in the WITH clause in a DDL statement that is used to create a Hologres table. You can use SQL hints to add or modify these parameters in the INSERT statement.
Realtime Compute for Apache Flink whose engine version is vvr-6.0.5-flink-1.15 or later allows you to configure column descriptions when you create a Hologres table. This way, the related physical table inherits the column descriptions of the Hologres table.
Select the table creation statement and click Run that appears on the left side of the code.
When you create a Hologres table, you can configure physical table properties in the WITH clause. You can configure table property settings based on your business requirements to sort and query data in an efficient manner. The following table describes the supported property parameters and examples. The parameters whose names start with the table_property. prefix indicate physical table properties.
Parameter | Example | Allow modification |
table_property.orientation | 'table_property.orientation' = 'row,column' | No |
table_property.table_group | 'table_property.table_group' = 'table_group_xxx' | |
table_property.distribution_key | 'table_property.distribution_key' = 'a,b' | |
table_property.clustering_key | 'table_property.clustering_key' = 'a,b:desc' | |
table_property.segment_key | 'table_property.segment_key' = 'c,d' | |
table_property.bitmap_columns | 'table_property.bitmap_columns' = 'a:on,b:off' | Yes |
table_property.dictionary_encoding_columns | 'table_property.dictionary_encoding_columns' = 'a:on,b:off,c:auto' | |
table_property.time_to_live_in_seconds | 'table_property.time_to_live_in_seconds' = '864000' | |
table_property.binlog.level | 'table_property.binlog.level' = 'replica' | |
table_property.binlog.ttl | 'table_property.binlog.ttl' = '86400' | |
enableTypeNormalization | 'enableTypeNormalization' = 'true' | Specifies whether to enable the type normalization mode when you use a Hologres catalog to create a table. Valid values:
Note
|
The table properties supported by Hologres catalogs are the same as the table properties supported by the Hologres service. The table_property.
prefix is added to the names of the properties for Hologres catalogs to distinguish between the table properties of the Hologres service and Hologres catalogs. For more information about the parameters, see CREATE TABLE and Subscribe to Hologres binary logs.
Modify a Hologres table
Only Realtime Compute for Apache Flink whose engine version is vvr-6.0.5-flink-1.15 or later allows you to modify a Hologres table.
The following examples show the modifications that you can make on a Hologres table.
Modify table properties
ALTER TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` SET ( 'table_property.binlog.level' = 'replica', 'table_property.binlog.ttl' = '64700' );
NoteYou can modify only some table properties. For more information, see Create a Hologres table.
Rename a table
ALTER TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` RENAME TO `${catalog_name}`.`${db_name}`.`${schema_name.new_table_name}`;
Add columns to a table
ALTER TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` ADD <column_name> <column_datatype> COMMENT '<column_comment>';
Modify the name of a column
ALTER TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` RENAME <old_column_name> TO <new_column_name>;
Modify the comment of a column
ALTER TABLE `${catalog_name}`.`${db_name}`.`${schema_name.table_name}` MODIFY <column_name> <original_column_type> COMMENT '<new_column_comment>';
Read data from a Hologres table
INSERT INTO ${other_sink_table}
SELECT ...
FROM `${catalog_name}`.`${db_name}`.`${schema_name.table_name}`
Insert result data into a Hologres table
INSERT INTO `${catalog_name}`.`${db_name}`.`${schema_name.table_name}`
SELECT ...
FROM ${other_source_table}
Use the Hologres catalog that you created as the catalog of the destination store that is used in the CREATE TABLE AS statement
CREATE TABLE IF NOT EXISTS `${catalog_name}`.`${db_name}`.`${schema_name.table_name}`
WITH (
'connector' = 'hologres'
) AS TABLE ${other_source_table};
The CREATE TABLE AS statement allows you to configure physical table attributes in the WITH clause. When you create a destination table, you can configure these attributes for the table. For more information about the table properties supported by Hologres catalogs, see Create a Hologres table.
To ensure that data can be written to Hologres when data is synchronized from the source table, the Hologres catalog is forced to modify the schema of the result table in the following scenarios:
A column whose data type is DECIMAL is used as the primary key in the schema of the source table.
Hologres does not support the primary keys of the DECIMAL type. Therefore, Hologres changes the data type of the column to BIGINT. If the change does not meet your business requirements, you can use the CREATE TABLE AS statement to convert the data type of the referenced column to the STRING type and recreate a primary key.
The schema of the source table contains values of the TIME, TIMESTAMP, or TIMESTAMP_LTZ type. The precision of these data types is greater than 6.
The precision of the time types supported by Hologres is 6. To ensure that data can be written to Hologres, Realtime Compute for Apache Flink implicitly discards the parts that are higher than the highest precision supported by Hologres.
Use the Hologres catalog that you created as the catalog of the destination store that is used in the CREATE DATABASE AS statement
CREATE DATABASE IF NOT EXISTS `${catalog_name}`.`${db_name}`
WITH (
'sink.parallelism' = '5' -- Specify the degree of parallelism for each result table.
) AS DATABASE ${other_source_database};
During data synchronization, you can declare the parameters of the result table in the WITH clause. When the job starts, these parameters take effect on the result tables to which you want to synchronize data. For more information about the parameters that can be modified, see Create a Hologres result table.
The CREATE DATABASE AS statement allows you to specify the schemaname parameter together with the parameters of the Hologres result table in the WITH clause. This way, data can be synchronized to the specified schema of the destination Hologres database. The following table describes the schemaname parameter.
Parameter | Description | Required | Remarks |
schemaname | The name of the schema. | No | Default value: public. |
You may need to configure different table properties for each destination table. You cannot configure properties for each table in the WITH clause. Therefore, you cannot execute the CREATE DATABASE AS statement to configure physical table properties. If you want to configure table properties, you must manually create a destination table before you start a CREATE DATABASE AS deployment. For more information about the table properties supported by Hologres catalogs, see the "Create a Hologres table" section.
Drop a Hologres catalog
Drop a Hologres catalog on the UI
Go to the Catalogs page.
Log on to the management console of Realtime Compute for Apache Flink.
Find the workspace that you want to manage and click Console in the Actions column.
In the left-side navigation pane, click Catalogs.
On the Catalog List page, find the desired catalog and click Delete in the Actions column.
In the message that appears, click Delete.
NoteAfter you drop the catalog, you can view the Catalogs pane on the left side of the Catalog List page to check whether the catalog is dropped.
Drop a Hologres catalog by executing an SQL statement
In the code editor of the Scripts tab on the SQL Editor page, enter the following statement:
DROP CATALOG ${catalog_name}
catalog_name is the name of the Hologres catalog that you want to drop from the development console of Realtime Compute for Apache Flink.
WarningThe drop operation does not affect the deployments that are running. However, deployments that are not published or deployments that you want to suspend and then resume are affected. Proceed with caution.
Right-click the statement that is used to drop the catalog and select Run from the shortcut menu.
View the Catalogs pane on the left side of the Catalog List page to check whether the catalog is dropped.