Apache Kyuubi is a distributed and multi-tenant gateway that provides query services such as SQL queries for data lake query engines. The data lake query engines include Spark, Flink, and Trino. DataWorks provides E-MapReduce (EMR) Kyuubi nodes that you can use to develop and periodically schedule Kyuubi tasks and integrate Kyuubi tasks with other types of tasks. This topic describes how to use an EMR Kyuubi node to develop tasks.
Prerequisites
An Alibaba Cloud EMR cluster is created and registered to DataWorks. For more information, see Register an EMR cluster to DataWorks.
(Required if you use a RAM user to develop tasks) The RAM user is added to the DataWorks workspace as a member and is assigned the Develop or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.
A serverless resource group is purchased and configured. The configurations include association with a workspace and network configuration. For more information, see Create and use a serverless resource group.
A workflow is created in DataStudio.
Development operations in different types of compute engines are performed based on workflows in DataStudio. Therefore, before you create a node, you must create a workflow. For more information, see Create a workflow.
Limits
This type of node can be run only on a serverless resource group or an exclusive resource group for scheduling. We recommend that you use a serverless resource group.
Step 1: Create an EMR Kyuubi node
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Create an EMR Kyuubi node.
Find the desired workflow, right-click the workflow name, and then choose .
NoteAlternatively, you can move the pointer over the Create icon and choose .
In the Create Node dialog box, configure the Name, Engine Instance, Node Type, and Path parameters. Click Confirm. The configuration tab of the EMR Kyuubi node appears.
NoteThe node name can contain only letters, digits, underscores (_), and periods (.).
Step 2: Develop an EMR Kyuubi task
You can develop a Kyuubi task on the configuration tab of the EMR Kyuubi node.
Develop SQL code
In the SQL editor, develop node code. You can define variables in the ${Variable} format in the node code and configure the scheduling parameters that are assigned to the variables as values in the Scheduling Parameter section of the Properties tab. This way, the values of the scheduling parameters are dynamically replaced in the node code when the node is scheduled to run. For more information about how to use scheduling parameters, see Supported formats of scheduling parameters. Sample code:
show tables;
select * from kyuubi040702 where age >= '${a}'; -- You can assign a specific scheduling parameter to the var variable.
The size of the SQL statements for the task cannot exceed 130 KB.
If multiple EMR data sources are associated with DataStudio in your workspace, you must select one from the data sources based on your business requirements. If only one EMR data source is associated with DataStudio in your workspace, you do not need to select a data source.
(Optional) Configure advanced parameters
You can configure advanced parameters on the Advanced Settings tab of the configuration tab of the current node. For more information about how to configure the parameters, see Spark Configuration.
Advanced parameter | Description |
queue | The scheduling queue to which jobs are committed. Default value: Note If a workspace-level YARN queue is configured when you register an EMR cluster to the desired workspace, the following rules that are used to select a scheduling queue apply when you run a Kyuubi task:
For information about EMR YARN, see YARN schedulers. For information about how to configure a queue when you register an EMR cluster, see Configure a global YARN queue. |
priority | The priority. Default value: 1. |
FLOW_SKIP_SQL_ANALYZE | The execution method of SQL statements. Valid values:
Note This parameter is available only for testing in the development environment of a DataWorks workspace. |
DATAWORKS_SESSION_DISABLE | Specifies whether to establish a JDBC connection every time an SQL statement is executed. This parameter is available for testing in the development environment of a DataWorks workspace. Valid values:
Note If the DATAWORKS_SESSION_DISABLE parameter is set to |
Run the Kyuubi task
In the toolbar, click the
icon. In the Parameters dialog box, select the desired resource group from the Resource Group Name drop-down list and click Run. NoteIf you want to access a data source over the Internet or a virtual private cloud (VPC), you must use the resource group for scheduling that is connected to the data source. For more information, see Network connectivity solutions.
If you want to change the resource group in subsequent operations, you can click the
(Run with Parameters) icon to change the resource group in the Parameters dialog box.
Click the
icon in the top toolbar to save the SQL statements. Optional. Perform smoke testing.
You can perform smoke testing on the node in the development environment when you commit the node or after you commit the node. For more information, see Perform smoke testing.
Step 3: Configure scheduling properties
If you want the system to periodically run a task on the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure task scheduling properties based on your business requirements. For more information, see Overview.
You must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
Step 4: Deploy the task
After a task on a node is configured, you must commit and deploy the task. After you commit and deploy the task, the system runs the task on a regular basis based on scheduling configurations.
Click the
icon in the top toolbar to save the task. Click the
icon in the top toolbar to commit the task. In the Submit dialog box, configure the Change description parameter. Then, determine whether to review task code after you commit the task based on your business requirements.
NoteYou must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
You can use the code review feature to ensure the code quality of tasks and prevent task execution errors caused by invalid task code. If you enable the code review feature, the task code that is committed can be deployed only after the task code passes the code review. For more information, see Code review.
If you use a workspace in standard mode, you must deploy the task in the production environment after you commit the task. To deploy a task on a node, click Deploy in the upper-right corner of the configuration tab of the node. For more information, see Deploy nodes.
What to do next
After you commit and deploy the task, the task is periodically run based on the scheduling configurations. You can click Operation Center in the upper-right corner of the configuration tab of the corresponding node to go to Operation Center and view the scheduling status of the task. For more information, see View and manage auto triggered nodes.