fully managed Flink - Hologres - Alibaba Cloud Documentation Center

Fully managed Flink is a real-time computing service provided by Alibaba Cloud. It is a fully managed service that is developed based on Apache Flink. Hologres is highly compatible with fully managed Flink. You can use Hologres source tables, dimension tables, result tables, and catalogs in fully managed Flink. This allows you to build a one-stop real-time data warehouse by using fully managed Flink. This topic describes how to use Hologres instances as upstream and downstream storage services to store data in fully managed Flink.

Hologres source tables, dimension tables, and result tables

Fully managed Flink supports Hologres source tables, dimension tables, and result tables. For more information, see Hologres connector.

Real-time consumption of binary logs

Realtime Compute for Apache Flink consumes binary log data in real time by using the Hologres connector. Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 6.0.3 or later supports streaming source tables in Java Database Connectivity (JDBC) mode. Realtime Compute for Apache Flink supports more data types in JDBC mode than those in HoloHub mode and supports custom accounts in JDBC mode. For more information, see Use Realtime Compute for Apache Flink to consume binary logs in real time.

Hologres Catalog

Realtime Compute for Apache Flink supports Hologres catalogs. You can read Hologres metadata in the development console of Realtime Compute for Apache Flink without the need to register Hologres tables. This improves deployment development efficiency and ensures table schema accuracy. For more information, see Manage Hologres catalogs.

Realtime Compute for Apache Flink supports schema evolution and database synchronization based on Hologres catalogs. For more information, see CREATE TABLE AS statement and CREATE DATABASE AS statement.

Hologres DataStream Connector

If you use Hologres DataStream connectors to read data from or write data to Hologres, you need to use Hologres DataStream connectors to connect Hologres to Realtime Compute for Apache Flink. For more information, see Hologres DataStream connector.

Mappings between data types

For more information about the mappings between data types in Realtime Compute for Apache Flink and Hologres, see Data types.

Note

When you deploy SQL drafts in Realtime Compute for Apache Flink, you must define data types to those that are supported by Realtime Compute for Apache Flink, regardless of whether you use Hologres source tables, dimension tables, or result tables in the SQL deployments. You can define data types to those that are supported by Hologres only when you create an internal table in Hologres.

If you want to use Realtime Compute for Apache Flink to write JSON data to Hologres, you must define the data types of columns for the source table and result table in the SQL deployment to the VARCHAR type, and define the data types of columns for the Hologres internal table to the JSONB type.

Hologres internal table: Set the data type for message to JSONB.

BEGIN ;
DROP TABLE IF EXISTS holo_internal_table;
CREATE TABLE IF NOT EXISTS holo_internal_table
(
    id BIGINT NOT NULL,
    message JSONB NOT NULL
);
CALL set_table_property('holo_internal_table', 'distribution_key', 'id');
COMMIT ;

Flink deployment: Set the data type for message in the source table and result table to VARCHAR.

CREATE TEMPORARY TABLE randomSource (
    id BIGINT,
    message VARCHAR
  )
WITH ('connector' = 'datagen');

CREATE TEMPORARY TABLE sink_holo (
    id BIGINT,
    message VARCHAR
  )
WITH (
    'connector' = 'hologres',
    'endpoint' = '',
    'username' = '',
    'password' = '',
    'dbname' = '',
    'tablename' = 'holo_internal_table'
  );

INSERT INTO sink_holo
SELECT
  1,
  '{"k":"v"}'
FROM
  randomSource;

Connector Release Note

For more information about the connector release notes, see Hologres connector release note.

FAQ

For more information about common issues and the solutions to the issues when you use Realtime Compute for Apache Flink, see Troubleshoot Blink and Flink issues.