Hologres is seamlessly integrated with MaxCompute at the underlying layer. Hologres can directly read data from MaxCompute, and MaxCompute can also directly read data from Hologres. This significantly improves data access efficiency. This topic describes the capabilities and limits of using Hologres to accelerate data reads from and data writes to MaxCompute.
Background information
MaxCompute is a fast and fully managed computing platform for large-scale data warehousing. It can process up to exabytes of data. MaxCompute is used to store and compute large amounts of structured data at a time. MaxCompute provides various data warehousing solutions and big data analysis and modeling services. For more information, see What is MaxCompute?
Hologres is a real-time interactive analysis engine that is compatible with PostgreSQL and uses the standard PostgreSQL protocol. Hologres is natively integrated with MaxCompute storage resources. In Hologres, you can perform accelerated queries on MaxCompute data by using external databases or foreign tables, without the need to store data in Hologres or import data to or export data from Hologres. You can use Hologres to connect to mainstream business intelligence (BI) tools.
Hologres outperforms other third-party big data services in accelerating data queries from MaxCompute, importing MaxCompute data, and writing data back to MaxCompute.
Scenarios
Accelerated queries on MaxCompute data
Hologres can directly read MaxCompute data that uses the Apsara Distributed File System storage. This method delivers better performance and is easier to use. You can execute the CREATE EXTERNAL DATABASE statement to create a database in Hologres to map schemas and tables in a MaxCompute project. You can also execute the IMPORT FOREIGN SCHEMA statement to create multiple foreign tables at a time. If you want to accelerate queries on a small number of MaxCompute tables or on specific fields in a MaxCompute table, you can execute the CREATE FOREIGN TABLE statement. Select a method based on your business characteristics and scenarios.
In versions earlier than Hologres V3.0, a maximum of 200 GB of MaxCompute data in a maximum of 512 partitions can be scanned at a time. In Hologres V3.0 and later, the limits are removed. Large-scale queries consume more resources. We recommend that you perform resource isolation. For more information, see Getting started with virtual warehouse instances. You can also use serverless computing resources to perform occasional large-scale queries. For more information, see Overview of Serverless Computing.
Data import from MaxCompute to Hologres for queries
To improve query performance, you can import data to Hologres and then query the data. In this method, you can leverage the index capabilities of Hologres to further accelerate queries. Complex queries, such as queries that involve the UPDATE, INSERT, and DELETE operations, are supported. For more information, see Import data from MaxCompute to Hologres by executing SQL statements.
Data export from Hologres to MaxCompute
For more information about how to write data that is written to Hologres in real time or data that is aggregated and analyzed in Hologres back to MaxCompute, see Export data to MaxCompute.
Description
The following table describes the capabilities provided by Hologres for accelerated queries on MaxCompute data by using foreign tables.
Hologres version | Supported capability | Description |
V3.0.7 | The CREATE EXTERNAL DATABASE statement is supported to create an external database in Hologres for mapping schemas and tables in a MaxCompute project. | You can execute the CREATE EXTERNAL DATABASE statement to create an external database in Hologres and use the external database to load the metadata of an external data source to Hologres. This allows you to manage internal and external data in Hologres and helps implement centralized metadata management by using the integrated lakehouse architecture. For more information, see CREATE EXTERNAL DATABASE. |
The limits that a maximum of 200 GB of MaxCompute data in a maximum of 512 partitions can be scanned by using a foreign table at a time are removed. | None. | |
V2.2.1 | The Auto Load feature supports MaxCompute projects that use the three-layer model. | Schemas in the three-layer model of MaxCompute are used to classify tables, resources, and functions in a project. For more information about schemas, see Schema. For more information about the Auto Load feature, see Auto Load. |
The Auto Load feature supports automatic updates of metadata. | None. | |
The Auto Load feature supports schema evolution on MaxCompute foreign tables. | Schema evolution on MaxCompute foreign tables includes operations such as deleting columns, changing the order of columns, and changing the data types of columns. For more information, see Auto Load. | |
The service-linked role of Hologres can be used to access MaxCompute. | A service-linked role is a RAM role whose trusted entity is an Alibaba Cloud service. Service-linked roles can implement authorized access across Alibaba Cloud services. For more information, see Service-linked role for Hologres. | |
V2.1 | You can write data to tables in a specified schema in a MaxCompute project that uses the three-layer model. | Schemas in the three-layer model of MaxCompute are used to classify tables, resources, and functions in a project. For more information about schemas, see Schema. |
You can access tables in a specified schema in a MaxCompute project that uses the three-layer model in the HoloWeb console. | For more information, see Create a foreign table mapped to MaxCompute. | |
V2.0 | The engine for accelerated queries on MaxCompute data is upgraded to Hologres Query Engine (HQE) to improve the compatibility and stability. | None. |
V1.3 | You can use foreign tables to query data in MaxCompute transactional tables. | None. |
You can read data from MaxCompute tables in the schema evolution state. | A table is in the schema evolution state if its columns are deleted, the order of the columns is changed, or the data types of the columns are changed in MaxCompute. | |
Dual-signature authentication between Hologres and MaxCompute is supported. | For more information, see Create a Hologres external table in dual-signature mode in the "Hologres external tables" topic. | |
You can access MaxCompute tables that are encrypted based on Bring Your Own Key (BYOK). | For more information, see Query MaxCompute data encrypted based on BYOK. | |
V1.1.43 | The Auto Load feature is supported. Foreign tables can be automatically created for queried MaxCompute tables based on your business requirements or all MaxCompute tables. | For more information, see Auto Load. |