Features and benefits of Realtime Compute for Apache Flink - Realtime Compute for Apache Flink

This topic describes the features and benefits of Realtime Compute for Apache Flink compared with the open source version of Apache Flink.

Category	Feature	Description	Benefit
Performance and cost	Full compatibility	Ensures complete compatibility with the open source version of Apache Flink, including support for APIs across various abstraction levels, parameter configurations, and SQL syntax.	Compared with Apache Flink, Realtime Compute for Apache Flink provides enhanced performance and fine-grained resource management to help you reduce the total cost of ownership (TCO). You can select a billing method based on your business requirements and use automatic scaling to optimize resource utilization.
	Enhanced performance	Adopts GeminiStateBackend as the backend storage system. GeminiStateBackend is developed by Alibaba Cloud and uses a new architecture to support compute-storage separation and key-value separation. This resolves the limitations of local disk capacity for state data and significantly enhances the efficiency of dual-stream and multi-stream join operations. GeminiStateBackend also supports adaptive parameter tuning, which eliminates the need to manually modify parameters. The Nexmark benchmark test results show that GeminiStateBackend provides twice the stream computing performance of Apache Flink. For more information, see GeminiStateBackend and Performance white paper (Nexmark performance testing). Adopts an enhanced SQL engine that is compatible with the syntax used in Apache Flink. The engine includes a series of enhancements, including an optimized state structure for operators, latency materialization at the computing layer, Codegen improvements, and improvements for join operations. For example, the engine supports caching of dimension tables, MiniBatch, data skew mitigation, and fine-grained state settings. The preceding enhancements help optimize CPU, memory, and state storage utilization.
	Efficient resource utilization	Supports scaling based on your business requirements. For more information, see Dynamically update the parameter configuration for dynamic scaling.
		Supports automatic tuning. The system monitors and adjusts deployment resource allocation and executes a resource plan at a specific point in time. This helps ensure stability during peak hours and maintains costs at a reasonable level. For more information, see Configure automatic tuning.
		Supports fine-grained resource management. You can configure CPU and memory resources at the SQL operator level. The resource utilization of large-scale deployments can be improved by 100%. For more information, see Configure resources for a deployment.
	Flexible billing methods	Supports the subscription and pay-as-you-go billing methods. You can select a method based on your business requirements. For more information, see Billable items.
Distinctive features	Real-time data ingestion	Supports real-time synchronization of data changes and schema changes in a database, including databases that use sharding or partitioning. For more information, see Data synchronization templates.	You can ingest data from databases or message-oriented middleware into data lakes or data warehouses in real time.
	Real-time fraud detection	Supports enterprise-class complex event processing (CEP). You can dynamically configure rules for a deployment without the need to restart the deployment, which ensures the continuous operational capability in scenarios such as online real-time fraud detection. For more information, see CEP statements.	You can enhance development efficiency and large-scale data processing while ensuring business continuity for mission-critical scenarios, such as real-time marketing, real-time fraud detection, and Threat Detection Service (TDS).
	Powerful connectors	Supports more than 30 mainstream systems provided by Alibaba Cloud and the Apache community, including databases, message-oriented middleware, data warehouses, data lakes, and file systems. For more information, see Supported connectors. Provides the Faker connector to help you generate test data that closely resembles real business data. Provides higher usability and stability than open source connectors. Supports custom connectors that you can use to access external storage systems.	Realtime Compute for Apache Flink is integrated with various upstream and downstream systems, which eliminates the need to build from scratch and ensures system stability and optimal performance.
Development efficiency	Coding	Provides an end-to-end development and management platform. You can use SQL, Java, Scala, and Python for development.	You do not need to build from scratch or customize the version provided by Apache Flink. Flink SQL is easy to use and simplifies development.
		Supports mainstream Apache Flink versions. You can compare the deployment code for different versions and roll back to a previous version. For more information, see Manage deployment versions.
		Supports centralized metadata management. You can use catalogs to connect common upstream and downstream systems, such as MySQL, Apache Hive, Hologres, Data Lake Formation (DLF), and Apache Kafka. For more information, see Manage catalogs.
		Supports user-defined functions (UDFs). You can easily manage and use UDFs. For more information, see Manage UDFs.
		Provides more than 20 Flink SQL templates for common scenarios to help you quickly get started. For more information, see Code templates.
	Debugging	Supports test data management. You can sample online data or generate simulated data to build testing workflows. For more information, see Debug a deployment.	Programmers and data analysts can easily debug and run a deployment. This significantly reduces costs and improves efficiency during the testing stage.
		Supports efficient debugging. You can start or cancel deployments in session clusters within seconds.
		Supports display of intermediate results, which improves the efficiency of debugging complex SQL statements.
		Supports isolation of the development and production environments. This ensures that debugging operations do not affect deployments and data in the production environment.
O&M	Monitoring and alerting	Provides various metrics and aggregation dimensions to help you detect performance issues, such as deployment latency, data skew, and backpressure. For more information, see Metrics.	You can achieve higher system stability with reduced O&M workload, simpler tuning operations, and significantly lower costs. You can also obtain high availability assurance provided by Alibaba Cloud.
	Monitoring and alerting	Sends alerts at the earliest opportunity to DingTalk or by email, text message, or call. You can also use Managed Service for Prometheus to receive alerts. For more information, see Report metrics of Realtime Compute for Apache Flink to other platforms.
	Issue analysis and diagnostics	Supports dynamic deployment configuration. For example, you can change the log level and enable or disable the flame graph feature without the need to cancel or restart the deployment.
	Issue analysis and diagnostics	Provides intelligent diagnostics for common issues, such as backpressure, deployment errors, and TaskManager disconnections. Issues are identified based on log analysis, and tuning and modification suggestions are automatically generated. For more information, see Perform intelligent deployment diagnostics.
	High availability	Guarantees a 99.9% service availability as indicated in the Service Level Agreement (SLA).
		Supports end-to-end automatic fault recovery, including fault tolerance for JobManagers. This prevents single points of failure and improves service stability.
		Supports fast single-node fault recovery to balance data consistency and service continuity.
	State management	Supports complete lifecycle management of system checkpoints and savepoints, state compatibility checks, and state data migration. This ensures maximum reuse of state data.
Enterprise security	Resource isolation	Provides tenant-level and project-level resource and code isolation. This ensures data security during cross-team collaboration.	Enterprises can perform secure and controlled inter-departmental collaboration based on internal and external audit standards.
Enterprise security	Access control	Supports access control for multiple users and roles based on the Alibaba Cloud account system.