Manage Simple Log Service catalogs - Realtime Compute for Apache Flink

You can configure a Simple Log Service catalog to enable the system to automatically parse logs in a Logstore of Simple Log Service to infer the table schema. After you configure a Simple Log Service catalog, you can directly access the Logstore of Simple Log Service on the SQL Editor page of the development console of Realtime Compute for Apache Flink. This way, you can obtain field information without the need to declare the schema of the Simple Log Service table in Flink SQL. This topic describes how to create, view, use, and drop a Simple Log Service catalog in the development console of Realtime Compute for Apache Flink.

Context

This topic describes the following operations that you can perform to manage Simple Log Service catalogs:

Create a Simple Log Service catalog
View a Simple Log Service catalog
View information about a Logstore of Simple Log Service
Use a Simple Log Service catalog
Drop a Log Service catalog

Limits

All fields in a table are parsed as strings. If you want to use other data types, you must use Flink SQL to perform data type conversion.
Only Realtime Compute for Apache Flink that uses Ververica Runtime (VVR) 6.0.7 or later supports Simple Log Service catalogs.
You cannot use DDL statements to modify existing Simple Log Service catalogs.
You can only query data tables by using Simple Log Service catalogs. You are not allowed to create, modify, or delete databases and tables by using Simple Log Service catalogs.
The tables provided by Simple Log Service catalogs can be directly used as source tables and result tables in Flink SQL deployments but cannot be used as lookup dimension tables.

Create a Simple Log Service catalog

You can create a Simple Log Service catalog on the UI or by executing an SQL statement. We recommend that you create a Simple Log Service catalog on the UI.

Create a Simple Log Service catalog on the UI

Go to the Catalogs page.
1. Log on to the Realtime Compute for Apache Flink console. Find the workspace that you want to manage and click Console in the Actions column.
2. In the left-side navigation pane, click Catalogs.
On the Catalog List page, click Create Catalog. In the Create Catalog dialog box, click SLS on the Built-in Catalog tab in the Choose Catalog Type step and click Next.

Configure the parameters in the Configure Catalog step.

Important

After you create a Simple Log Service catalog, the parameter configuration cannot be modified. If you want to modify the parameter configuration, you must drop the Simple Log Service catalog that you created and create a Simple Log Service catalog again.

Parameter	Data type	Description	Required	Remarks
catalog name	STRING	The name of the Simple Log Service catalog.	Yes	Enter a custom name.
endpoint	STRING	The endpoint of Simple Log Service.	Yes	For more information, see Endpoints.
project	STRING	The name of the Simple Log Service project.	Yes	N/A.
accessId	STRING	The AccessKey ID of the Alibaba Cloud account.	Yes	For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account? Important To protect your AccessKey pair, we recommend that you configure the AccessKey ID by using the key management method. For more information, see Manage keys.
accessKey	STRING	The AccessKey secret of your Alibaba Cloud account.	Yes	For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account? Important To protect your AccessKey pair, we recommend that you configure the AccessKey secret by using the key management method. For more information, see Manage keys.

Click Confirm.
View the catalog that you create in the Catalogs pane on the left side of the Catalog List page.

Create a Simple Log Service catalog by executing an SQL statement

In the code editor of the Scripts tab on the SQL Editor page, enter the following statement to create a Simple Log Service catalog:

CREATE CATALOG <catalogName> WITH(
  'type'='sls',
  'endpoint'='<brokers>',
  'project'='project',
  'accessId'='${secret_values.ak_id}',
  'accessKey'='${secret_values.ak_secret}'
)

Parameter	Data type	Description	Required	Remarks
catalogName	STRING	The name of the Simple Log Service catalog.	Yes	Enter a custom name.
type	STRING	The type of the catalog.	Yes	Set the value to sls.
endpoint	STRING	The endpoint of Simple Log Service.	Yes	For more information, see Endpoints.
project	STRING	The name of the Simple Log Service project.	Yes	N/A.
accessId	STRING	The AccessKey ID of the Alibaba Cloud account.	Yes	For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account? Important To protect your AccessKey pair, we recommend that you configure the AccessKey ID by using the key management method. For more information, see Manage keys.
accessKey	STRING	The AccessKey secret of your Alibaba Cloud account.	Yes	For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account? Important To protect your AccessKey pair, we recommend that you configure the AccessKey secret by using the key management method. For more information, see Manage keys.

Select the code that is used to create a catalog and click Run that appears on the left side of the code. You can also right-click the code that is used to create the catalog and select Run.
In the Catalogs pane on the left side of the Catalog List page, view the catalog that you create.

View a Simple Log Service catalog

After a Simple Log Service catalog is configured, you can perform the following steps to view the metadata of Simple Log Service.

Go to the Catalogs page.
1. Log on to the Realtime Compute for Apache Flink console.
2. Find the workspace that you want to manage and click Console in the Actions column.
3. In the left-side navigation pane, click Catalogs.
On the Catalog List page, find the desired catalog and view the Name and Type columns of the catalog.
Note
If you want to view information about a Logstore in a catalog, click View in the Actions column of the catalog.

View information about a Logstore of Simple Log Service

In the code editor of the Scripts tab on the SQL Editor page, enter the following statement:

DESCRIBE `${catalog_name}`.`${project_name}`.`${logstore_name}`;

Parameter	Description
${catalog_name}	The name of the Simple Log Service catalog.
${project_name}	The name of the Simple Log Service project.
${logstore_name}	The name of the Logstore of Simple Log Service.

Select the code that is used to view a catalog and click Run on the left side of the code.
After the code is run, you can view the information about the table in the result.

Use a Simple Log Service catalog

Use a table of a Simple Log Service catalog as a source table and read data from the Logstore of Simple Log Service.
```
INSERT INTO ${other_sink_table}
SELECT...
FROM `${catalog_name}`.`${project_name}`.`${logstore_name}`/*+OPTIONS('startTime'='2023-06-01 00:00:00')*/;
```
Note
If you want to specify other parameters in the WITH clause when you use a table of a Simple Log Service catalog, we recommend that you use SQL hints to add the parameters. For example, the SQL hints in the preceding SQL statement specify that data consumption starts from '2023-06-01 00:00:00'. For more information about other parameters, see Simple Log Service connector.
Use a table of a Simple Log Service catalog as a result table and write data to the Logstore of Simple Log Service.
```
INSERT INTO `${catalog_name}`.`${project_name}`.`${logstore_name}`
SELECT ... 
FROM ${other_source_table}
```

Important

Before you use a Simple Log Service catalog to read data from the source Logstore, you must enable indexing. Similarly, you must enable indexing before you use a Simple Log Service catalog to write data to the destination Logstore. This is because a catalog needs to read specific data from the result table first to determine whether the schema of the data that needs to be written is the same as the schema of the destination Logstore of Simple Log Service. For more information about how to enable indexing, see Create indexes.

Drop a Simple Log Service catalog

You can drop a Simple Log Service catalog on the UI or by executing an SQL statement. We recommend that you drop a Simple Log Service catalog on the UI.

Warning

After you drop a Simple Log Service catalog, the deployments that are running are not affected. However, the deployments that use a table of the catalog can no longer find the table if the deployments are published or restarted. Proceed with caution when you drop a Simple Log Service catalog.

Drop a Simple Log Service catalog on the UI

Go to the Catalogs page.
1. Log on to the Realtime Compute for Apache Flink console.
2. Find the workspace that you want to manage and click Console in the Actions column.
3. In the left-side navigation pane, click Catalogs.
On the Catalog List page, find the desired catalog and click Delete in the Actions column.
In the message that appears, click Delete.
View the Catalogs pane on the left side of the Catalog List page to check whether the catalog is dropped.

Drop a Simple Log Service catalog by executing an SQL statement

In the code editor of the Scripts tab on the SQL Editor page, enter the following statement:
```
DROP CATALOG ${catalog_name};
```
In the preceding statement, ${catalog_name} indicates the name of the Simple Log Service catalog that you want to delete.
Right-click the statement that is used to drop the catalog and select Run from the shortcut menu.
View the Catalogs pane on the left side of the Catalog List page to check whether the catalog is dropped.

Information about a table that is obtained by using a Simple Log Service catalog

When you use a Simple Log Service catalog to obtain data from a Simple Log Service table, the default parameters and metadata are added to the derived table. This helps you use the table in an efficient manner. A table that is obtained by using a Simple Log Service catalog contains the following information:

Schema inference of the related Simple Log Service table
When a Simple Log Service catalog parses logs in Log Simple Service to obtain the schema of a topic, the catalog attempts to consume a message and parse the message to obtain the schema of the data. The Simple Log Service catalog parses logs in Simple Log Service to obtain the field names and data types. The logs in Simple Log Service are stored as data of the STRING type. Therefore, all fields are of the STRING type.

Table parameters that need to be added by default

Parameter	Description	Remarks
connector	The type of the connector.	Set the value to sls.
endpoint	The endpoint of Simple Log Service.	For more information, see Endpoints.
project	The name of the Simple Log Service project.	N/A.
logstore	The name of a Logstore or Metricstore of Simple Log Service.	N/A.
accessId	The AccessKey ID of the Alibaba Cloud account.	For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account? Important To protect your AccessKey pair, we recommend that you configure the AccessKey ID by using the key management method. For more information, see Manage keys.
accessKey	The AccessKey secret of your Alibaba Cloud account.	For more information, see How do I view information about the AccessKey ID and AccessKey secret of the account? Important To protect your AccessKey pair, we recommend that you configure the AccessKey secret by using the key management method. For more information, see Manage keys.