Realtime Compute for Apache Flink:Service notices of Realtime Compute (Blink) (phased out)

Among all Blink 2.X versions, Blink 2.2.7 is the latest stable version. Compared with Blink 1.X versions, Blink 2.2.7 has significant improvements. Among all Blink 1.X versions, Blink 1.6.4 is the latest stable version. Blink 2.2.7 uses the new generation of Niagara as the data storage system for the state backend platform. Blink 2.2.7 also optimizes SQL performance and introduces a wide range of new features.

• Features

  • SQL

    • Supports the EMIT statement for windows. The EMIT statement is used to define the output policy. For example, you can view the latest stream processing result every minute in a one-hour window.

    • Supports miniBatch for joining two streams. Blink 2.2.7 optimizes the retraction mechanism and state data storage structure to improve performance.

    • Supports data filtering in aggregation. You can aggregate the rows that meet specified filter conditions.

    • Optimizes local-global aggregation.

    • Restructures the SQL code during the optimization stage. This shortens the time to compile code.

    • Allows keys in SortedMapView to support multiple data types, such as BOOLEAN, BYTE, SHORT, INT, LONG, FLOAT, DOUBLE, BIGDECIMAL, BIGINT, BYTE[], and STRING.

    • Optimizes the performance in the scenarios in which retractions are involved in the MIN, MAX, FIRST, or LAST function.

    • Adds multiple scalar functions, including the following functions that are related to time zones: TO_TIMESTAMP_TZ, DATE_FORMAT_TZ, and CONVERT_TZ.

    • Classifies SQL and connector error messages into different types and provides an error code for each type of error message.

  • Connector

    • Allows you to register the connectors of source tables and result tables by using a custom TableFactory.

    • Allows you to parse data source types by using user-defined table-valued functions (UDTFs).

    • Allows you to read data from and write data to Kafka.

    • Allows you to write data to Elasticsearch.

  • Runtime

    • Streamlines behavior such as submitting jobs and obtaining job running results by using Blink sessions.

    • Supports the plug-in scheduling mechanism. Based on this mechanism, a computing model can customize the scheduling logic to meet your business requirements.

    • Optimizes the failover-triggered restart policy. In throttling scenarios, you can restart only the required tasks in a failed job for recovery instead of restarting all tasks in the job. This optimization improves the efficiency of JobManagers and task failover processing.

  • StateBackend

    • Replaces RocksDBStateBackend with NiagaraStateBackend to improve read/write performance.

    • (Experimental) NiagaraStateBackend separates computing from storage. This state backend helps you restore state data in a few seconds if failovers occur.

• Items incompatible with Blink 1.6.4

  Feature	Impact	Solution
  TableFunction interface changes	The changes affect the users who use UDTFs.	Update the code to implement the getResultType interface.
  ScalarFunction interface changes	The changes affect the users who use user-defined scalar functions.	Implement the getResultType interface.
  AggregateFunction interface changes	The changes affect the users who use user-defined aggregation functions (UDAFs).	Implement the getAccumulatorType and getResultType interfaces. For example, if the accumulator is of the Row(STRING, LONG) type and the Agg result is of the STRING type, use the following code: public DataType getAccumulatorType\(\) { return DataTypes.createRowType\(DataTypes.String, DataTypes.LONG\); } public DataType getResultType\(\) { return DataTypes.String; }
  MapView constructor changes	The parameter that indicates the data type is changed from TypeInformation to DataType. The change affects the jobs that declare the MapView in the UDAFs.	Update the code and create a MapView by using the DataType parameter. For example, before the change, the code is MapView map = new MapView<>(Types.STRING, Types.INT);. After the change, the code is MapView map = new MapView<>(DataTypes.STRING, DataTypes.INT);.
  The type of the values returned by the AVG and division functions is changed to DOUBLE if the input parameter is of the LONG or INT type	Before the change, the AVG and division functions return the same type of data as the input parameters. After the change, the data type of the return data is DOUBLE. This may cause type mismatch errors. For example, if the data returned by the AVG or division functions is written to a result table, an error may occur. This is because the type of the data in the result table is different from the data type of the query fields.	Add the CAST function to perform explicit conversion on the data that is returned by the AVG and division functions.
  Precision is considered when data of the BIGDECIMAL type and data of the DECIMAL type are compared	For jobs that use the DECIMAL data type, a type mismatch error may occur. This is because the data type of the actual data is BIGDECIMAL.	Specify the precision when you declare the DECIMAL data type in the job code: Decimal(38, 18).
  Semantics used to compare between the null value and strings	In Blink 1.X versions, the SQL statement returns true if a string is null. In Blink 2.X versions, the SQL statement returns false to comply with SQL syntax and semantics.	The change affects processing in scenarios in which the comparison between the null value and strings is involved. For example, if SPLIT_INDEX(shop_list, ':', 1) in the WHERE SPLIT_INDEX(shop_list, ':', 1) <> shop_id clause returns null, the WHERE clause returns true in Blink 1.X versions and false in Blink 2.X versions. This facilitates data filtering.

• Update to Blink 2.X

  To update a Blink 1.X version to a Blink 2.X version for a job, you must reset a start offset when you restart the job to meet your business requirements.

  1. Terminate the job. After the job is terminated, the state data is cleared.

  2. In the lower-right corner of the Development page, select Blink 2.2.7 from the Flink Version drop-down list. Then, publish the job.

  3. Start the job on the Administration page and specify the start offset.

     If Step 3 fails, perform the following steps for troubleshooting:

     • Verify and fix the SQL code, and repeat Steps 1 to 3.

     • If the SQL code cannot be fixed, roll back the Blink version to the original version.

     • If a JSON plan cannot be generated, specify the following parameters:

       • blink.job.option.jmMemMB=4096

       • blink.job.submit.timeoutInSeconds=600

  For information about the JAR packages for the user-defined extensions (UDXs) in Blink 2.0.1, see Overview. The following errors may occur because the version of the UDX JAR package is earlier than the required version or a package conflict occurs:

  code:[30016], brief info:[get app plan failed], context info:[detail:[java.lang.NoClassDefFoundError: org/apache/flink/table/functions/aggfunctions/DoubleSumWithRetractAggFunction
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:788)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:73)