Realtime Compute for Apache Flink:GeminiStateBackend

Stateful computing is a complex and challenging scenario in stream processing. Data access for stream processing has the following characteristics:

• A large number of random accesses exist and few range queries are performed.

• Data traffic and hotspots frequently change. In this case, different parallel threads of the same operator use different data access modes.

GeminiStateBackend provides the following benefits:

• Uses a new architecture and data structure design to improve the overall data processing performance.

  The overall architecture of GeminiStateBackend is designed based on the log-structured merge-tree (LSM tree) data structure. GeminiStateBackend provides three capabilities: adaption with changes in the data volume and access characteristics, tiered storage of hot and cold data, and switchover between anti-caching and caching architectures. GeminiStateBackend also supports the hash storage structure that allows random access. The performance comparison by using Nexmark shows that GeminiStateBackend provides better performance than RocksDBStateBackend. The performance of about half of the GeminiStateBackend use cases is more than 70% higher than the performance of the RocksDBStateBackend use cases.

• Supports compute-storage separation to eliminate the dependency of state data on local disks.

  The space of local disks is limited. Therefore, a deployment that has a large amount of state data often encounters the issue of insufficient local disk space. In most cases, if a deployment that runs based on RocksDBStateBackend encounters the issue of insufficient local disk space, you need to increase the parallelism of threads or use other methods to increase resources. GeminiStateBackend supports compute-storage separation. This way, state storage can be independent of local disks. This prevents deployment failures that are caused by excessive local state data. For more information about the configuration related to compute-storage separation, see Parameters for compute-storage separation.

• Supports adaptive key-value separation to significantly improve the performance of deployments that involve dual-stream JOIN or multi-stream JOIN.

  Dual-stream JOIN or multi-stream JOIN is one of the most challenging scenarios in stream processing and a typical scenario in which state storage encounters a bottleneck. GeminiStateBackend provides the key-value separation feature to adapt to scenarios where the success rate of JOIN operations is low or the length of state data values is long. This significantly improves the performance of deployments that involve dual-stream JOIN or multi-stream JOIN. The key-value separation feature supports adaptive adjustment without the need for manual configurations and optimization. The verification during Double 11 Shopping Festival of Alibaba Group shows that the throughput of deployments can be increased by 50% to 70% or a higher percent and the computing resource utilization can be increased by an average of 50% after the key-value separation feature is enabled. In typical scenarios, the computing resource utilization can be increased by 100% to 200%. For more information about the configuration related to key-value separation, see Parameters for key-value separation.

• Uses lightweight savepoints, which significantly accelerates the creation of checkpoints and savepoints for deployments that contain a large amount of state data.

  GeminiStateBackend supports finer-grained deployment savepoints and decouples the checkpoint feature from the compaction mechanism of the LSM tree. This accelerates and stabilizes the creation of checkpoints and savepoints. GeminiStateBackend also supports the Native Incremental Savepoint feature. This feature is used together with native savepoints of Realtime Compute for Apache Flink to provide performance that is similar to the performance of checkpoints. This improves the availability of savepoints.

• Supports adaptive parameter tuning, which prevents the workloads of manual parameter tuning.

  In stream processing tasks, different operators have different state access modes. In most cases, different combinations of parameters are required to implement the optimal performance of state storage. The configurations of these parameters involve underlying technologies. Manual parameter tuning has high learning and understanding costs. To resolve this issue, GeminiStateBackend supports the adaptive parameter tuning technology. When a deployment is running, parameter configurations can be automatically tuned based on the current data access mode and traffic to achieve the optimal performance of state storage in various scenarios. The verification during Double 11 Shopping Festival of Alibaba Group shows that this technology can reduce manual parameter tuning by more than 95% and increase the single-core throughput by 10% to 40%. For more information about the configuration related to adaptive parameter tuning, see Parameters for adaptive parameter tuning.

In this example, the use cases related to state bottlenecks and hardware resources in Nexmark are used to compare the performance between RocksDBStateBackend and GeminiStateBackend.

The comparison result shows that GeminiStateBackend significantly optimizes the overall performance (single-core throughput) of deployments. The following table shows the comparison result.