This topic describes the major updates and bug fixes of the Realtime Compute for Apache Flink version released on July 22, 2024.
The version upgrade is incrementally rolled out across the network by using a canary release strategy. For information about the upgrade schedule, see the latest announcement on the right side of the management console of Realtime Compute for Apache Flink. You can use the new features in this version only after the upgrade is complete for your account. To apply for the upgrade at the earliest opportunity, submit a ticket.
Overview
This release includes platform and engine updates.
Platform updates
Platform updates in this release focus on ease of use, system stability, security, and O&M efficiency.
Custom roles and fine-grained permissions: Custom roles can be created and fine-grained permissions can be assigned based on your business requirements. This improves the flexibility of permission management and the security of job-related operations.
More efficient development and O&M: The UI for job details and resource configuration is optimized to improve clarity and ease of use. MaxCompute catalogs can be created on the GUI, which facilitates the configuration and management of Flink jobs and improves data management and development efficiency.
Engine updates
Ververica Runtime (VVR) 8.0.8 is officially released to provide an enterprise-class engine based on Apache Flink 1.17.2. VVR 8.0.8 includes the following updates:
Connectors: (1) The handling policy of RETRACT events and the maximum number of returned documents can be configured for the Elasticsearch connector. (2) Deletion policies can be separately configured for different columns in a table created by using the Hologres connector. (3) The JSON data type is supported by the StarRocks connector. (4) The latest and earliest startup modes are supported by the Simple Log Service (SLS) connector. (5) ApsaraDB RDS for MySQL endpoints can be configured for the MySQL connector.
SQL: The URL_DECODE and URL_ENCODE functions are added as built-in functions.
Performance: The performance of Apache Paimon dimension tables during join operations is enhanced. The speed of writing full and incremental data to Hologres by using MySQL Change Data Capture (CDC) is improved. The speed of reading data from MaxCompute source tables is improved, and the data writing workload on Flink clusters is reduced.
Security: Passwords are encrypted in the job topology to prevent exposure of sensitive information.
For information about the major updates in this release and the related references, see the next section of this topic. The version upgrade is incrementally rolled out across the network by using a canary release strategy. After the upgrade is complete for your account, we recommend that you upgrade the VVR engine to this version. For more information, see Upgrade the engine version of a deployment. We look forward to your feedback.
Features
Feature | Description | References |
Custom roles | Custom roles can be created and fine-grained permissions can be assigned based on your business requirements. This enhances the flexibility of permission management and the security of job-related operations. | |
UI optimization related to job details | The parameter configuration used to start a job can be viewed by performing the following steps: Go to the deployment details page, click the Status tab, and then click Job Details in the Actions section. | N/A |
UI optimization related to resource configuration | The maximum parameter values are displayed in the Resources section of the Configuration tab. | N/A |
JSON support by the StarRocks connector | The JSON data type is supported. This resolves the exception that occurs when JSON data from MySQL is written to StarRocks. | |
Enhanced Elasticsearch connector |
| |
Startup mode for SLS source tables | SLS data can be consumed from the latest or earliest offset. | |
New Docker image of PyFlink | The basic Docker image of PyFlink is upgraded to improve compatibility with different Python and glibc versions. | N/A |
URL_DECODE and URL_ENCODE functions | Built-in functions for URL encoding and decoding are added to streamline implementation. | |
Configuration of ApsaraDB RDS for MySQL endpoints | Binary logs in the associated Object Storage Service (OSS) bucket can be read by using MySQL CDC. | |
Partial update configuration for the Hologres connector | Deletion policies can be separately configured for different columns in a table created by using the Hologres connector. For example, you can configure the table to delete entire rows. You can also set the values of non-primary-key columns to Null, which ensures that other columns are not affected in partial update scenarios. | |
Custom partitioner for joining dimension tables | The shuffle strategy for join operations that involve dimension tables can be configured to improve data processing efficiency. | |
Enhanced MaxCompute connector | The Arrow format can be used to read data, and shards can be dynamically allocated. This improves read performance. |
Fixed issues
If data from multiple streams is written to an Apache Paimon table during partial updates, conflicts occur.
If out-of-memory (OOM) errors occur when MySQL CDC is used to parse a large number of binary logs, the system performs only silent retries. Starting from this release, the system throws an exception and triggers a job failover.
If OOM errors related to GeminiStateBackend occur, the system performs only silent retries. Starting from this release, the system throws an exception and triggers a job failover.
The consumption status of the ApsaraMQ for RocketMQ connector is not displayed in the ApsaraMQ for RocketMQ console.
If the schema of a Hologres source table changes in a job that is started based on state data, snapshots fail to be restored. For example, TRUNCATE operations change the table schema.
The java.lang.NoClassDefFoundError: StringUtils error may occur when you use the StarRocks connector.
All issues in Apache Flink 1.17.2. For more information, see Apache Flink 1.17.2 Release Announcement.