This topic describes the major updates and fixed issues of the Realtime Compute for Apache Flink version released on October 23, 2023.
Ververica Runtime (VVR) 8.0.3 introduced in this release ocassionally causes data loss in specific scenarios, which affects data accuracy. After careful evaluation, Alibaba Cloud decides to announce the End of Support (EOS) for VVR 8.0.3. We recommend that you upgrade to VVR 8.0.5 or later at the earliest opportunity. For information about how to upgrade, see Upgrade the engine version of deployments. We provide necessary support and guidance to help you smoothly transition to a more secure and stable version. Thank you for your understanding and cooperation.
Overview
The release of Realtime Compute for Apache Flink on October 23, 2023 includes platform updates, engine updates, connector updates, performance optimization, and bug fixes.
Engine updates: VVR 8.0.3 is officially released to provide an enterprise-class engine based on Apache Flink 1.17.1. In VVR 8.0.3, the interoperability between Realtime Compute for Apache Flink and other Alibaba Cloud storage and computing services is enhanced. The MaxCompute, StarRocks, ApsaraDB for Redis, Simple Log Service, and ApsaraMQ for RocketMQ connectors are optimized to improve performance and stability. The usability of the CREATE TABLE AS statement is also improved.
Platform updates: The stable strategy is introduced in the Autopilot tuning mode to prevent frequent deployment restarts. Multiple scheduled tuning plans can be configured for a deployment, and the plan in use can be changed without the need to cancel the deployment. Field-level data lineage is supported for SQL deployments to help you manage real-time data assets. Dynamic parameter updates are supported to reduce the number of deployment restarts. Logon by using a Resource Access Management (RAM) role, resource directory member, or CloudSSO user is supported.
The DataStream APIs for the preceding mentioned connectors will be released in the next version.
The version upgrade is rolled out across the network by using a canary release strategy during a two-week period. After the upgrade is complete for your region and account, you can use the new engine version for your deployments. For more information, see Upgrade the engine version of deployments. We look forward to your feedback.
Features
Feature | Description | References |
Optimized automatic tuning | The stable strategy is supported to help identify the optimal resource configuration for the entire runtime of a deployment. The system automatically adjusts the resource configuration of a deployment only if the system finds a better option for the entire runtime of the deployment. This prevents unnecessary modification due to sudden traffic bursts and allows the deployment to run in a stable manner and reach the convergence state. | |
Scheduled tuning plans | Multiple scheduled tuning plans can be configured for a deployment, and the plan in use can be changed without the need to cancel the deployment. One scheduled tuning plan can contain multiple scheduled tuning strategies. | |
Data lineage of Flink SQL deployments | The data lineage of Flink SQL deployments can be viewed. This feature allows you to find deployments that use a specific field in a table and identify the field-level relationships between the source and result tables in a deployment. This way, you can efficiently manage deployments and data assets. | |
Dynamic updates of parameter configurations | The configuration of the Parallelism parameter and specific runtime parameters of a Flink deployment can be dynamically modified without the need to cancel the deployment. | Dynamically update the parameter configuration for dynamic scaling |
SQL optimization suggestions | Optimization suggestions and potential risks can be viewed when you perform a syntax check on an SQL deployment. You can optimize the SQL statements based on this information. | |
Label-based search | A label can be specified when you create a SQL, JAR, or Python deployment. After you specify a label for a deployment, you can search for all deployments that use the same label key or label value on the Deployments page in the development console of Realtime Compute for Apache Flink. This helps you manage deployments in an efficient manner. | |
Enhanced deployment sorting and filtering | Deployments can be sorted by health score or latency and filtered based on the user who perform a modification. This feature is available on the Deployments page in the development console of Realtime Compute for Apache Flink. | N/A |
Logon support for RAM roles, resource directory members, and CloudSSO users | The management and development consoles of Realtime Compute for Apache Flink are seamlessly integrated with the Alibaba Cloud account system. You can log on to the consoles by using an Alibaba Cloud account, a RAM user, a RAM role, a resource directory member, or a CloudSSO user. This simplifies identity management and access control and enhances authorization and resource management. | |
Support for complex MaxCompute data types and exclusive MaxCompute Tunnel | Data of the JSON, ARRAY, MAP, and STRUCT types used in MaxCompute can be read and written by the MaxCompute connector. An exclusive Tunnel can be specified for the MaxCompute connector to facilitate connection between MaxCompute and Realtime Compute for Apache Flink. | |
New data caching policy in ApsaraDB for Redis dimension tables | The cache parameter can be set to ALL when you use the ApsaraDB for Redis connector to create a dimension table. This improves performance. | |
Enhanced compatibility with ApsaraDB RDS for MySQL | ApsaraDB RDS for MySQL instances for which transparent data encryption (TDE) is enabled are supported by MySQL catalogs. | N/A |
Support for OSS-HDFS endpoints in Data Lake Formation (DLF) catalogs | The oss.endpoint parameter can be set to an OSS-HDFS endpoint for a DLF catalog. | |
Type normalization support for schema changes when a result table is created by using the StarRocks connector in the CREATE TABLE AS statement | The type normalization mode for schema changes is supported when you use the StarRocks connector in the CREATE TABLE AS statement to synchronize data to a result table. After you enable this feature, if a field in the source table is changed to a data type that has a different length but is compatible with the data type of the corresponding field in the result table, synchronization is not affected. | N/A |
DATE columns as partition keys in Hologres result tables | A column of the DATE data type can be used as a partition key. The partitioned table is automatically created. | |
Support for ApsaraMQ for RocketMQ 5.x | The ApsaraMQ for RocketMQ connector can be used to read data from and write data to ApsaraMQ for RocketMQ 5.x tables. | ApsaraMQ for RocketMQ connector |
Enhanced miniBatch and join performance | The miniBatch feature and the performance of join operations are optimized. If you perform a join operation in Change Data Capture (CDC) scenarios or after operations such as deduplication and aggregation, the throughput can be increased by up to 130%. | N/A |
Fixed issues
An error occurs when a MaxCompute catalog is used to write data to a partitioned table.
The performance of Hologres dimension tables is poor during join operations.