Supported connectors

This topic describes the types of tables and the connectors that are supported by Realtime Compute for Apache Flink.

Table types

Alibaba Cloud Realtime Compute for Apache Flink allows you to use Flink SQL to define a table that provides the mappings between the upstream and downstream storage, or use the DataStream API to access the upstream and downstream storage to perform read and write operations. Realtime Compute for Apache Flink supports the following types of tables that are defined by using Flink SQL:

Source table

A source table serves as a data input table of a Realtime Compute for Apache Flink deployment and triggers a stream processing.
A source table cannot be used as a dimension table and must be used as a driving table to promote subsequent computing. The computing results of a source table trigger the computing chain.
In most cases, a source table can be used for conversion and computing of up to tens of millions or even hundreds of millions of data records.
A source table provides streaming data inputs to trigger and push the data processing of Realtime Compute for Apache Flink. The continuous stream of new data can be from storage such as a message queue service or database change logs.
A source table contains key fields that can be joined and associated, such as user ID and order ID as primary keys.

Dimension table

A dimension table is an auxiliary table, which is used as an extension of the data of a source table.
A dimension table cannot be used as a driving table. It can only be used as a supplement for a source table. A dimension table does not drive computing.
In most cases, a dimension table can store only a small amount of data, such as gigabytes or terabytes of data. A dimension table can be a static table or a low-throughput streaming table.
A dimension table provides supplemental information to business data, such as user names, product details, and region information.
A dimension table can be joined with a source table to enrich the data of the source table to form a more detailed wide table.

Sink table

A sink table is an output data table of a Realtime Compute for Apache Flink deployment.
A sink table stores the final result data after computing and conversion, such as aggregation results and data obtained after filtering.
The final result data that is stored in a sink table can be exported to external systems, such as databases, message queues, and file systems, for subsequent analysis.
A sink table is the final output of the entire deployment processing chain. It stores the output of the computing.

For example, the following source table and dimension table are provided for data analysis in a deployment:

Source table: an order table that contains columns such as user ID, order ID, and order amount.
Dimension table: a user information table that contains columns such as user ID, user name, and address. This is a static table.

When the deployment runs, Realtime Compute for Apache Flink reads real-time order data from the source table and joins the order data stream with the user information in the dimension table. Realtime Compute for Apache Flink aggregates data to obtain the total order amount by region and writes the aggregation results to the sink table.

In this deployment, the order table is used as the source table, the user information table is used as the dimension table, and the statistical sink table is used as the final output of the deployment. The order table cannot be used as a dimension table and must be used as a driving table for data input. The user information table cannot be used as a driving table and can only be used as a dimension table to provide additional order data.

Connector	Table type			Running mode	API type	Data update or deletion in a sink table
Connector	Source table	Dimension table	Sink table	Running mode	API type	Data update or deletion in a sink table
Kafka connector	√	×	√	Streaming mode	SQL API, DataStream API, and YAML API for data ingestion	Data in a sink table cannot be updated or deleted. Data can only be inserted into a sink table.
Hologres connector	√	√	√	Streaming mode and batch mode	SQL API, DataStream API, and YAML API for data ingestion	Supported.
Simple Log Service connector	√	×	√	Streaming mode	SQL API and DataStream API	Data in a sink table cannot be updated or deleted. Data can only be inserted into a sink table.
MySQL connector Note The MySQL connector supports ApsaraDB RDS for MySQL, PolarDB for MySQL, and self-managed MySQL databases.	√	√	√	Streaming mode	SQL API, DataStream API, and YAML API for data ingestion	Supported.
ApsaraDB RDS for MySQL connector Note The ApsaraDB RDS for MySQL connector will not be supported in the future. We recommend that you use the MySQL connector instead of the ApsaraDB RDS for MySQL connector.	×	√	√	Streaming mode and batch mode	SQL API	Supported.
MaxCompute connector	√	√	√	Streaming mode and batch mode	SQL API and DataStream API	Data in a sink table cannot be updated or deleted. Data can only be inserted into a sink table.
Tair (Redis OSS-compatible) connector	×	√	√	Streaming mode	SQL API	Supported.
Upsert Kafka connector	√	×	√	Streaming mode	SQL API and YAML API for data ingestion	Supported.
Elasticsearch connector	√	√	√	Streaming mode and batch mode	SQL API and DataStream API	Supported.
AnalyticDB for MySQL V3.0 connector	×	√	√	Streaming mode and batch mode	SQL API	Supported.
ClickHouse connector	×	×	√	Streaming mode and batch mode	SQL API	Supported.
Print connector	×	×	√	Streaming mode and batch mode	SQL API and YAML API for data ingestion	Supported.
Blackhole connector	×	×	√	Streaming mode and batch mode	SQL API	Supported.
ApsaraDB for HBase connector	×	√	√	Streaming mode	SQL API	Supported.
Datagen connector	√	×	×	Streaming mode and batch mode	SQL API	N/A.
ApsaraMQ for RocketMQ connector	√	×	√	Streaming mode	SQL API and DataStream API	Data in a sink table cannot be updated or deleted. Data can only be inserted into a sink table.
Tablestore connector	√	√	√	Streaming mode	SQL API	Supported.
JDBC connector	√	√	√	Streaming mode and batch mode	SQL API	Supported.
SelectDB connector (public preview)	×	×	√	Streaming mode and batch mode	DataStream API and SQL API	Supported.
StarRocks connector	√	×	√	Streaming mode and batch mode	SQL API, DataStream API, and YAML API for data ingestion	Supported.
PostgreSQL CDC connector (public preview)	√	×	×	Streaming mode	SQL API	N/A.
AnalyticDB for PostgreSQL connector	×	√	√	Streaming mode and batch mode	SQL API	Supported.
Lindorm connector	×	√	√	Streaming mode	SQL API	Supported.
OSS connector	√	×	√	Streaming mode and batch mode	SQL API and DataStream API	Data in a sink table cannot be updated or deleted. Data can only be inserted into a sink table.
Faker connector	√	√	×	Streaming mode and batch mode	SQL API	N/A.
Apache Iceberg connector	√	×	√	Streaming mode and batch mode	SQL API	Supported.
TSDB for InfluxDB connector	×	×	√	Streaming mode	SQL API	Not supported.
Apache Paimon connector	√	√	√	Streaming mode and batch mode	SQL API and YAML API for data ingestion	Supported.
Hudi connector	√	×	√	Streaming mode and batch mode	SQL API and DataStream API	Supported.
Tair (Enterprise Edition) connector	×	×	√	Streaming mode	SQL API	Supported.
OceanBase connector (public preview)	√	√	√	Streaming mode and batch mode	SQL API	Supported.
MongoDB connector	√	√	√	Streaming mode	SQL API and DataStream API	Supported.
PolarDB for Oracle 1.0 connector	×	×	√	Streaming mode and batch mode	SQL API	Supported.