Release notes of Realtime Compute for Apache Flink (November 21, 2024) - Realtime Compute for Apache Flink

This topic describes the major updates and bug fixes of the Realtime Compute for Apache Flink version released on November 21, 2024.

Important

The version upgrade is incrementally rolled out across the network by using a canary release plan. For information about the upgrade schedule, see the latest announcement on the right side of the Realtime Compute for Apache Flink console. You can use the new features in this version only after the upgrade is complete for your account. To apply for the upgrade at the earliest opportunity, submit a ticket.

Overview

This release includes updates of the platform, engine, and connectors, as well as performance optimization and bug fixes.

Platform updates

Platform updates in this release focus on ease of use, system stability, and O&M efficiency. The highlights are as follows:

Support for hybrid billing: This new billing method combines the benefits of subscription and pay-as-you-go billing. It allows you to configure both fixed and elastic resources for your Flink workspace. Hybrid billing not only reduces the need to allocate excessive resources for unpredictable or temporary workload spikes, but also ensures a fixed amount of resources is reliably at your disposal. You can use this billing method along with the platform's automatic tuning feature for further cost optimization.
Console homepage redesign: The updated homepage provides easy access to frequently used features and overviews of resources and deployments.
Support for renaming existing drafts.
Improved version management: As Flink releases new versions, older versions gradually reach their End of Support (EOS). To ensure a stable and smooth version upgrade, your most recently used EOS engine versions are retained. This allows you to roll back when necessary.

Engine updates

Ververica Runtime (VVR) 8.0.10 is officially released to provide an enterprise-class engine based on Apache Flink 1.17.2. It also provides optimizations and enhancements beyond the latest bug fixes in Apache Flink. VVR 8.0.10 includes the following updates:

New capabilities

JDK 11 support: A new JDK version becomes available. However, compatibilities are not guaranteed between the use of JDK 11 or JDK 8 in different minor VVR versions.
SelectDB connector (public preview): This connector allows for writing data to ApsaraDB for SelectDB.

Note

These capabilities are currently in the experimental stage, and as such, their service level agreements (SLAs) are not guaranteed. Exercise caution when considering using them in production systems.

Enhanced capabilities

Enhanced SQL semantics: The processing-time temporal join is supported to correlate rows in a fact table to the latest version of a corresponding key in a dimension table. Unlike an event-time temporal join that correlates rows based on the time when events occurred, a processing-time temporal join correlates rows based on the data arrival time.
Newly supported built-in SQL function: PERCENTILE(expr, percentage[, frequency]) is supported.
CDC ingestion in YAML deployments: The Kafka connector can now be used as a source in a YAML deployment, supporting both Debezium JSON and Canal JSON formats. The Paimon and StarRocks connectors can handle upstream TRUNCATE and DROP TABLE statements. DECIMAL or TIMESTAMP columns with different precisions can be merged in a sharded database.
Enhanced access control for CREATE TABLE AS SELECT (CTAS) / CREATE DATABASE AS SELECT (CDAS): CTAS/CDAS statements support Apache Paimon catalogs of the DLF 2.0 metadata storage type, which enhances access control.
Enhanced StarRocks connector: BIGINT UNSIGNED and VARBINARY data types are supported.

Performance improvement

Faster database and incremental data ingestion: Unified batch and stream processing significantly boosts the performance of real-time ingestion of full and incremental data from MySQL CDC to Paimon.
Fully-managed storage optimization: In a workspace configured with fully managed storage, resuming from a savepoint requires less time and resources, which leads to increased cost optimization.

Experience optimization

Connector configuration experience: System configurations for certain Debezium-related options in the MySQL CDC connector take precedence over user configurations to prevent potential misuse. The timeout option of Hologres connector is optimized to reduce retry attempts during draft deployment, facilitating data writing to Hologres.
SQL draft development experience: During SQL draft validation, tips and suggestions about the use of the SinkMaterializer operator are optimized.

Security enhancement

Apache Paimon catalog: After an Apache Paimon catalog is created, the fs.oss.accessKeySecret parameter value is displayed as ciphertext to protect data security.

For more information about the major updates in this release and the related references, see the next section of this topic. The version upgrade is incrementally rolled out using a canary release plan. After the upgrade is complete for your account, we recommend that you upgrade the VVR engine to this version. For more information, see Upgrade the engine version of a deployment. We look forward to your feedback.

Features

Feature	Description	References
JDK 11 support	VVR 8.0.10 fully supports JDK 11. This helps developers optimize their Java applications with the new features and gives developers more choices for runtime environments, increasing platform flexibility and compatibility.	Develop a JAR draft Develop a Python API draft UDSFs
Dimension table joins in Keyed-Ordered mode	Key-Ordered mode is introduced for use cases where data is retrieved asynchronously from an external system and processed in the UpsertKey order. This mode effectively makes up for the deficiencies of Ordered and Unordered modes.	Key parameters
Enhanced CDC data ingestion in YAML deployments	The YAML deployment now supports using the Kafka connector as a source, enhancing the flexibility of Flink systems or applications that use YAML deployments to process Kafka data streams.	Kafka connector Develop a YAML draft for data ingestion (public preview)
Optimized SLS connector	The backoff policy is adopted to improve the SLS connector's stability and reliability.	N/A
Enhanced StarRocks connector	BIGINT UNSIGNED and VARBINARY data types are supported. The length of a CHAR column can be automatically extended to three times the length before mapping. This adapts to the differences in encoding between MySQL and StarRocks.	StarRocks
Enhanced SQL semantics	The processing-time temporal join is supported. It uses a processing-time attribute to correlate rows in a fact table to the latest version of a corresponding key in a dimension table.	Processing-time temporal join statements
New built-in SQL function	The PERCENTILE function is now supported.	Supported functions
Optimized Hive catalogs	You can create Hive catalogs in a workspace with fully managed storage, upload configuration files, and manage their lifecycles.	Manage Hive catalogs
Enhanced access control for CTAS/CDAS	CTAS/CDAS supports Apache Paimon catalogs of the DLF 2.0 metastore type.	CREATE TABLE AS statement CREATE DATABASE AS statement
Console UI optimization	The console home page is redesigned to provide overviews of resources and deployments, easy access to frequently used features, and navigation to relevant documentation.	N/A
Hybrid billing	Hybrid billing provides the flexibility of pay-as-you-go and cost-effectiveness of subscription.
Optimized log archiving	Expired archived logs are periodically cleared up to save storage costs.	View the logs of a historical deployment
SelectDB connector	ApsaraDB for SelectDB is a next-generation real-time data warehouse service. It is fully managed and hosted on Alibaba Cloud and 100% compatible with Apache Doris. Data can be written to ApsaraDB for SelectDB via the SelectDB connector.	SelectDB connector (public preview)

Fixed issues

Connector issues

MySQL CDC: Fixed an issue where data loss may occur during the transition from full data reading to binlog-based incremental reading through Object Storage Service (OSS).
Tair (Redis OSS-compatible): Fixed an issue where data cannot be written to Redis due to a deficit in Buffered Writer of the Tair (Redis OSS-compatible) connector in VVR 8.0.9.
OSS: Fixed a performance issue of writing data to OSS in VVR 8.0.7 or later.
Paimon: Fixed an issue where a time zone of the timestamp type is not properly converted in a YAML deployment.

SQL issues

Source merging: Fixed an issue where a deployment cannot be started when table.optimizer.source-merge.enabled is set to true.
Minibatch interval: Fixed an issue in VVR 8.0.7 where the minibatch interval configuration does not take effect.

Compatibility and dependency issues

Connector class loading: Fixed an issue where the exception connector class not found is reported when starting a deployment that uses a built-in connector with a JAR dependency.
Local run of IntelliJ IDEA: Fixed an issue where the error ClassNotFoundException MySqlSourceReaderMetrics is reported when running a deployment that uses a MySQL CDC JAR package locally in IntelliJ IDEA.

Dynamic configuration issues

Fixed an issue where dynamic updates do not take effect occasionally.