Realtime Compute for Apache Flink：Manage Apache Paimon catalogs

You can use SQL statements to create Apache Paimon catalogs of all the preceding metadata storage types. However, only Apache Paimon catalogs of the filesystem and dlf metadata storage types can be created in the console. This section describes the creation of Apache Paimon catalogs of different metadata storage types and configuration parameters.

• Create an Apache Paimon Filesystem catalog

• Create an Apache Paimon DLF catalog

• Create an Apache Paimon MaxCompute catalog

• Create an Apache Paimon Sync catalog

Create an Apache Paimon Filesystem catalog

Create an Apache Paimon DLF catalog in the console

1. Go to the Catalogs page.

   1. Log on to the Realtime Compute for Apache Flink console. Find the workspace that you want to manage and click Console in the Actions column.

   2. In the left-side navigation pane, click Catalogs.

2. Click Create Catalog. On the Built-in Catalog tab of the wizard that appears, choose Apache Paimon and click Next.

3. In the Configure Catalog step, configure the parameters.

Create an Apache Paimon catalog through SQL

Execute the following SQL statement in the SQL editor. For detailed instructions, see Scripts.

CREATE CATALOG `my-catalog` WITH (
  'type' = 'paimon',
  'metastore' = 'filesystem',
  'warehouse' = '<warehouse>',
  'fs.oss.endpoint' = '<fs.oss.endpoint>',
  'fs.oss.accessKeyId' = '<fs.oss.accessKeyId>',
  'fs.oss.accessKeySecret' = '<fs.oss.accessKeySecret>'
);

The following table describes the parameters in the SQL statements.

• Common parameters

  Parameter	Description	Required	Remarks
  my-catalog	The name of the Apache Paimon catalog.	Yes	Enter a custom name.
  type	The type of the catalog.	Yes	Set the value to paimon.
  metastore	The metadata storage type.	Yes	Valid values: filesystem: Creates an Apache Paimon Filesystem catalog. dlf: Creates an Apache Paimon DLF catalog. maxcompute: Creates an Apache Paimon MaxCompute catalog. sync: Creates an Apache Paimon Sync catalog.

• OSS

  Parameter	Description	Required	Remarks
  warehouse	The data warehouse directory that is specified in OSS.	Yes	Format: oss://<bucket>/<object>. Field description: bucket: the name of the OSS bucket that you created. object: the path in which your data is stored. You can view the values of the bucket and object fields in the OSS console.
  fs.oss.endpoint	The endpoint of OSS.	Yes	If DLF resides in the same region as your Realtime Compute for Apache Flink workspace, use the VPC endpoint. If they are not in the same region, use the public endpoint. These parameters are required if the OSS bucket specified by the warehouse parameter does not reside in the same region as the Realtime Compute for Apache Flink workspace or if an OSS bucket within another Alibaba Cloud account is used. For more information about how to obtain the required information, see Regions, endpoints, and open ports and Create an AccessKey pair.
  fs.oss.accessKeyId	The AccessKey ID of the Alibaba Cloud account or RAM user that has the read and write permissions on OSS.	Yes
  fs.oss.accessKeySecret	The AccessKey secret of the Alibaba Cloud account or RAM user that has the read and write permissions on OSS.	Yes

Create an Apache Paimon DLF catalog

CREATE CATALOG `my-catalog` WITH(
'metastore'='dlf-paimon',
'type'='paimon',
'dlf.endpoint'='<dlf.endpoint>',
'dlf.region'='<dlf.region>',
'dlf.catalog.instance.id'='<dlf.catalog.instance.id>'
);

CREATE CATALOG `my-catalog` WITH (
  'type' = 'paimon',
  'metastore' = 'dlf',
  'warehouse' = '<warehouse>',
  'dlf.catalog.id' = '<dlf.catalog.id>',
  'dlf.catalog.accessKeyId' = '<dlf.catalog.accessKeyId>',
  'dlf.catalog.accessKeySecret' = '<dlf.catalog.accessKeySecret>',
  'dlf.catalog.endpoint' = '<dlf.catalog.endpoint>',
  'dlf.catalog.region' = '<dlf.catalog.region>',
  'fs.oss.endpoint' = '<fs.oss.endpoint>',
  'fs.oss.accessKeyId' = '<fs.oss.accessKeyId>',
  'fs.oss.accessKeySecret' = '<fs.oss.accessKeySecret>'
);

Create an Apache Paimon MaxCompute catalog

• Prerequisites

  The Apache Paimon plug-in file is uploaded to your MaxCompute project by using one of the following methods:

  • Use the MaxCompute client (odpscmd) to access the MaxCompute project, and run the ADD JAR <path_to_paimon_maxcompute_connector.jar>; command to upload the Apache Paimon plug-in file to the MaxCompute project.

  • Create a resource in the DataWorks console to upload the Apache Paimon plug-in file to the MaxCompute project. For more information, see Create and use MaxCompute resources.

• SQL statements:

  CREATE CATALOG `my-catalog` WITH (
    'type' = 'paimon',
    'metastore' = 'maxcompute',
    'warehouse' = '<warehouse>',
    'maxcompute.endpoint' = '<maxcompute.endpoint>',
    'maxcompute.project' = '<maxcompute.project>',
    'maxcompute.accessid' = '<maxcompute.accessid>',
    'maxcompute.accesskey' = '<maxcompute.accesskey>',
    'maxcompute.oss.endpoint' = '<maxcompute.oss.endpoint>',
    'fs.oss.endpoint' = '<fs.oss.endpoint>',
    'fs.oss.accessKeyId' = '<fs.oss.accessKeyId>',
    'fs.oss.accessKeySecret' = '<fs.oss.accessKeySecret>'
  );

  Note

  When you create an Apache Paimon table in the Apache Paimon MaxCompute catalog, an Apache Paimon external table is automatically created in the MaxCompute project. To query the Apache Paimon external table in MaxCompute, you must execute the following SET statements, and then execute the SELECT statement in MaxCompute. For more information, see the "Step 4: Read data from the Apache Paimon external table on the MaxCompute client (odpscmd) or by using a tool that can execute MaxCompute SQL statements" section of the Apache Paimon external tables topic.

  SET odps.sql.common.table.planner.ext.hive.bridge = true;
  SET odps.sql.hive.compatible = true;

• The following table describes the parameters in the SQL statements.

  • Common parameters

    Parameter	Description	Required	Remarks
    my-catalog	The name of the Apache Paimon catalog.	Yes	Enter a custom name.
    type	The type of the catalog.	Yes	Set the value to paimon.
    metastore	The metadata storage type.	Yes	Valid values: filesystem: Creates an Apache Paimon Filesystem catalog. dlf: Creates an Apache Paimon DLF catalog. maxcompute: Creates an Apache Paimon MaxCompute catalog. sync: Creates an Apache Paimon Sync catalog.

  • OSS

    Parameter	Description	Required	Remarks
    warehouse	The data warehouse directory that is specified in OSS.	Yes	Format: oss://<bucket>/<object>. Field description: bucket: the name of the OSS bucket that you created. object: the path in which your data is stored. You can view the values of the bucket and object fields in the OSS console.
    fs.oss.endpoint	The endpoint of OSS.	Yes	If DLF resides in the same region as your Realtime Compute for Apache Flink workspace, use the VPC endpoint. If they are not in the same region, use the public endpoint. These parameters are required if the OSS bucket specified by the warehouse parameter does not reside in the same region as the Realtime Compute for Apache Flink workspace or if an OSS bucket within another Alibaba Cloud account is used. For more information about how to obtain the required information, see Regions, endpoints, and open ports and Create an AccessKey pair.
    fs.oss.accessKeyId	The AccessKey ID of the Alibaba Cloud account or RAM user that has the read and write permissions on OSS.	Yes
    fs.oss.accessKeySecret	The AccessKey secret of the Alibaba Cloud account or RAM user that has the read and write permissions on OSS.	Yes

  • MaxCompute

    Parameter	Description	Required	Remarks
    maxcompute.endpoint	The endpoint of MaxCompute.	Yes	For more information, see Endpoints.
    maxcompute.project	The name of the MaxCompute project.	Yes	MaxCompute projects with the schema feature enabled are not supported.
    maxcompute.accessid	The AccessKey ID of the Alibaba Cloud account that has permissions on MaxCompute.	Yes	For more information about how to obtain the AccessKey ID, see Create an AccessKey pair.
    maxcompute.accesskey	The AccessKey secret of the Alibaba Cloud account that has permissions on MaxCompute.	Yes	For more information about how to obtain the AccessKey secret, see Create an AccessKey pair.
    maxcompute.oss.endpoint	The endpoint that is used to access OSS from MaxCompute.	No	If you do not specify this parameter, the value of the fs.oss.endpoint parameter is used by default. For more information, see OSS regions and endpoints. Note We recommend that you set the maxcompute.oss.endpoint parameter to an VPC endpoint because the OSS bucket resides in the same region as the MaxCompute project.
    maxcompute.life-cycle	The lifecycle of the MaxCompute external table.	No	Unit: day.

Create an Apache Paimon Sync catalog

CREATE CATALOG `my-catalog` WITH (
  'type' = 'paimon',
  'metastore' = 'sync',
  'source' = 'dlf',
  'target' = 'maxcompute',
  'warehouse' = '<warehouse>',
  'dlf.catalog.id' = '<dlf.catalog.id>',
  'dlf.catalog.accessKeyId' = '<dlf.catalog.accessKeyId>',
  'dlf.catalog.accessKeySecret' = '<dlf.catalog.accessKeySecret>',
  'dlf.catalog.endpoint' = '<dlf.catalog.endpoint>',
  'dlf.catalog.region' = '<dlf.catalog.region>',
  'maxcompute.endpoint' = '<maxcompute.endpoint>',
  'maxcompute.project' = '<maxcompute.project>',
  'maxcompute.accessid' = '<maxcompute.accessid>',
  'maxcompute.accesskey' = '<maxcompute.accesskey>',
  'maxcompute.oss.endpoint' = '<maxcompute.oss.endpoint>',
  'fs.oss.endpoint' = '<fs.oss.endpoint>',
  'fs.oss.accessKeyId' = '<fs.oss.accessKeyId>',
  'fs.oss.accessKeySecret' = '<fs.oss.accessKeySecret>'
);

The following table describes the parameters in the SQL statements.

• Common parameters

  Parameter	Description	Required	Remarks
  my-catalog	The name of the Apache Paimon catalog.	Yes	Enter a custom name.
  type	The type of the catalog.	Yes	Set the value to paimon.
  metastore	The metadata storage type.	Yes	Valid values: filesystem: Creates an Apache Paimon Filesystem catalog. dlf: Creates an Apache Paimon DLF catalog. maxcompute: Creates an Apache Paimon MaxCompute catalog. sync: Creates an Apache Paimon Sync catalog.

• Sync exclusive

  Parameter	Description	Required	Remarks
  source	The storage service from which metadata is synchronized.	Yes	Set the value to dlf.
  target	The storage service to which metadata is synchronized.	Yes	Set the value to maxcompute.

• OSS

  Parameter	Description	Required	Remarks
  warehouse	The data warehouse directory that is specified in OSS.	Yes	Format: oss://<bucket>/<object>. Field description: bucket: the name of the OSS bucket that you created. object: the path in which your data is stored. You can view the values of the bucket and object fields in the OSS console.
  fs.oss.endpoint	The endpoint of OSS.	Yes	If DLF resides in the same region as your Realtime Compute for Apache Flink workspace, use the VPC endpoint. If they are not in the same region, use the public endpoint. These parameters are required if the OSS bucket specified by the warehouse parameter does not reside in the same region as the Realtime Compute for Apache Flink workspace or if an OSS bucket within another Alibaba Cloud account is used. For more information about how to obtain the required information, see Regions, endpoints, and open ports and Create an AccessKey pair.
  fs.oss.accessKeyId	The AccessKey ID of the Alibaba Cloud account or RAM user that has the read and write permissions on OSS.	Yes
  fs.oss.accessKeySecret	The AccessKey secret of the Alibaba Cloud account or RAM user that has the read and write permissions on OSS.	Yes

• DLF

  Parameter	Description	Required	Remarks
  dlf.catalog.id	The ID of the DLF data directory.	Yes	You can view the ID of the data directory in the DLF console.
  dlf.catalog.accessKeyId	The AccessKey ID that is used to access the DLF service.	Yes	For more information about how to obtain an AccessKey ID, see Create an AccessKey pair.
  dlf.catalog.accessKeySecret	The AccessKey secret that is used to access the DLF service.	Yes	For more information about how to obtain an AccessKey secret, see Create an AccessKey pair.
  dlf.catalog.endpoint	The endpoint of DLF.	Yes	For more information, see Supported regions and endpoints. Note If DLF resides in the same region as your Realtime Compute for Apache Flink workspace, use the VPC endpoint. If they are not in the same region, use the public endpoint.
  dlf.catalog.region	The region in which DLF is deployed.	Yes	For more information, see Supported regions and endpoints. Note Make sure that the value of this parameter matches the endpoint that is specified by the dlf.catalog.endpoint parameter.

• maxcompute

  Parameter	Description	Required	Remarks
  maxcompute.endpoint	The endpoint of MaxCompute.	Yes	For more information, see Endpoints.
  maxcompute.project	The name of the MaxCompute project.	Yes	MaxCompute projects with the schema feature enabled are not supported.
  maxcompute.accessid	The AccessKey ID of the Alibaba Cloud account that has permissions on MaxCompute.	Yes	For more information about how to obtain the AccessKey ID, see Create an AccessKey pair.
  maxcompute.accesskey	The AccessKey secret of the Alibaba Cloud account that has permissions on MaxCompute.	Yes	For more information about how to obtain the AccessKey secret, see Create an AccessKey pair.
  maxcompute.oss.endpoint	The endpoint that is used to access OSS from MaxCompute.	No	If you do not specify this parameter, the value of the fs.oss.endpoint parameter is used by default. For more information, see OSS regions and endpoints. Note We recommend that you set the maxcompute.oss.endpoint parameter to an VPC endpoint because the OSS bucket resides in the same region as the MaxCompute project.
  maxcompute.life-cycle	The lifecycle of the MaxCompute external table.	No	Unit: day.

Manage an Apache Paimon database

You can manage an Apache Paimon database by executing the following commands on the SQL Editor page. For more information, see Scripts.

• Create a database

  After you create an Apache Paimon catalog, a database named default is automatically created in the catalog.

  -- Replace my-catalog with the name of the actual Apache Paimon catalog. 
  USE CATALOG `my-catalog`;
  
  -- Replace my_db with a custom database name. 
  CREATE DATABASE `my_db`;

• Drop a database

  Important

  You cannot drop the default database from an Apache Paimon DLF catalog, an Apache Paimon MaxCompute catalog, or an Apache Paimon Sync catalog. You can only drop the default database from an Apache Paimon Filesystem catalog.

  -- Replace my-catalog with the name of the actual Apache Paimon catalog. 
  USE CATALOG `my-catalog`;
  
  -- Replace my_db with the name of the database that you want to drop. 
  DROP DATABASE 'my_db'; -- Drop an empty database. 
  DROP DATABASE `my_db` CASCADE; -- Drop the database and all the associated tables.

View or drop an Apache Paimon catalog

1. In the Realtime Compute for Apache Flink console, find the workspace that you want to manage and click Console in the Actions column.

2. In the left-side navigation pane, click Catalogs. A list of Apache Paimon catalogs displays.

   • View an Apache Paimon catalog: In the Catalog List section, find the catalog that you want to manage, and you can view its Name and Type. To see the databases and tables within the catalog, click View in the Actions column.

   • Drop an Apache Paimon catalog: In the Catalog List section, find the catalog that you want to manage and click Delete in the Actions column.

     Note

     After the Apache Paimon catalog is deleted, only the catalog information on the Catalogs page in the Flink namespace is deleted. The data files of the Apache Paimon tables remain. After the Apache Paimon catalog is deleted, you can re-create the Apache Paimon catalog by executing an SQL statement. Then, you can use the Apache Paimon tables in the catalog again.

     You can also drop an Apache Paimon catalog by executing the DROP CATALOG <catalog name>; command on the SQL Editor page. For more information, see Scripts.