Background information

As cloud storage, especially object storage, evolves, data lake solutions gradually evolve towards cloud-native technologies. OSS is used as the unified storage of cloud data lakes in the Alibaba Cloud lakehouse architecture to provide a secure, cost-effective, highly reliable, and scalable data lake solution.

The real-time data lake solution marks a significant development of data lakes. This solution focuses on the real-time and streaming performance of data in the lakehouse architecture. Hologres supports real-time data writes, real-time data updates, and real-time data analysis. Based on these powerful engine capabilities, Hologres integrates with Data Lake Formation (DLF), Hive Metastore Service (HMS), OSS, and various ecosystem capabilities to provide a comprehensive real-time data lake solution. The solution accelerates reads and writes of various types of data in OSS by using foreign tables without the need to migrate data. Foreign tables are used to map fields rather than storing data. This reduces development and O&M costs, breaks down data silos, and achieves business insights.

The following table describes the Alibaba Cloud services involved in the real-time data lake solution.

Architecture

The following figure shows the recommended data lake architecture for Hologres. The architecture covers the entire data lifecycle from collection, storage, and management to application. Hologres leverages powerful engine capabilities and flexible auto scaling policies to provide an end-to-end solution that integrates data lakes and warehouses.

Usage notes

• In Hologres V1.1 and later, you can read data in the ORC, Parquet, CSV, and SequenceFile formats from OSS. In Hologres V1.3 and later, you can write data in the ORC, Parquet, CSV, or SequenceFile format to OSS, and read data from Apache Hudi tables or Delta Lake tables in OSS.

  Note

  You can view the version of your Hologres instance on the instance details page in the Hologres console. If the version of your Hologres instance is earlier than V1.1, manually upgrade your Hologres instance in the Hologres console or join the Hologres DingTalk group to apply for an instance upgrade. For more information about how to manually upgrade a Hologres instance, see Instance upgrades. For more information about how to join the Hologres DingTalk group, see Obtain online support for Hologres.

• In Hologres V1.3.25 and later, you can use the multi-catalog feature of DLF to isolate metadata in the test environment, development environment, and cross-department instances. This helps ensure the security of your business. For more information about the multi-catalog feature, see Catalog.

• In Hologres V1.3.26 and later, you can read data from and write data to OSS-HDFS. The service capabilities and boundaries of data lake acceleration are further expanded. Hologres is seamlessly integrated with compute engines in the Hadoop ecosystem. This accelerates reads and writes of data stored in OSS-HDFS, greatly improves the real-time analysis efficiency of data in the Hadoop ecosystem, and meets the requirements of federated queries of data lakes and real-time data analysis in fields such as big data and AI.

• In Hologres V2.1.0 and later, you can read data from Apache Paimon foreign tables. Apache Paimon is a unified lake storage platform that allows you to process data in streaming and batch modes. Apache Paimon supports data writes with high throughput and data queries with low latency to allow data to flow in data lakes in real time. Users can integrate real-time and offline processing of data in data lakes by using Apache Paimon. For more information, see Apache Paimon.

• In Hologres V2.2 and later, the new foreign table architecture is used. In this architecture, Hologres Query Engine (HQE) allows you to directly read data from files in the ORC and Parquet formats, and use cache-based acceleration based on local SSDs. The performance is improved by more than five times. You can use HMS to access data in OSS and OSS-HDFS. For more information, see Use HMS to access data in OSS data lakes (beta).

  Note

  If the version of your Hologres instance is V2.1 or earlier, contact Hologres technical support to upgrade your instance.

• In Hologres V3.0 and later, the following features are added:

  • The external database feature is added to support catalog-level metadata mapping for data sources such as DLF and MaxCompute. This feature improves the metadata and data management capabilities of data lakes. For more information, see CREATE EXTERNAL DATABASE.

  • The external schema and external table capabilities are supported. You can create databases and tables in a specific DLF catalog to facilitate data write-back after aggregation. For more information, see CREATE EXTERNAL SCHEMA and CREATE EXTERNAL TABLE.

  • High-performance data writes to Apache Paimon append-only tables are supported to facilitate data forwarding in data lakes and data warehouses.

  • Paimon deletion vectors can be optimized to improve query performance when a large amount of data is deleted but compaction is not performed at the earliest opportunity.

  • Delta Lake readers are reconstructed to significantly improve read performance.

  • Data can be read from Iceberg-based data lakes. This helps expand the data lake ecosystem.

  • Data queries in EMR clusters are accelerated after Hologres connects to the HMS for metadata mapping. For more information, see Use the HMS to access data in OSS data lakes (beta).

  • Security capabilities are enhanced. By default, the service-linked role is used to access DLF 2.0. You can also use a RAM role to access DLF 2.0.

Scenarios

The following table describes the methods provided by Hologres to map external data sources.

Table formats and file formats

Table formats

File formats

Data type mappings

For more information about the mappings of data types between DLF and Hologres, see Data types.