MaxCompute Delta Table is an incremental table format. It enables minute-level data imports, uses AliORC as its base file format, and supports UPSERT operations. It provides a standardized Change Data Capture (CDC) method for reading and writing incremental data. The Delta Table automatically manages data based on the storage services and metadata services of MaxCompute. This topic describes the key functionalities of the MaxCompute Delta Table.
Delta Table format description
Table data format supports efficient storage and access of both incremental and full data. When a primary key is defined, the format enables real-time data updates.
Near-real-time incremental import
Full import and near-real-time incremental import for various data sources: MaxCompute is used together with related services to provide various custom data import tools, such as Use Flink to write data to a Delta table of MaxCompute and Data Integration of DataWorks. These tools support efficient near-real-time incremental import. These tools can be connected to the Near-real-time incremental import of the Tunnel service of MaxCompute to support high-concurrency minute-level incremental write.
Full write and incremental batch write for various data sources: MaxCompute SQL and other interfaces can be used to support efficient full write and incremental batch write.
Data organization optimization
The storage service of MaxCompute manages incremental data files in an intelligent manner. The management of incremental data files includes optimization operations, such as Clustering of small files, data COMPACTION, and data sorting. The storage service automatically executes data optimization tasks for some operations based on a comprehensive evaluation of multiple dimensions, such as data characteristics and time series, to help maintain stable and efficient data storage and computing.
Transaction management
This module is used for Manage transactions, data version management, time travel management, metadata updates, and metadata analysis.
Basic operations
DDL operation - Data Definition Language
Write data