Presto (namely PrestoDB) is a flexible and scalable distributed SQL query engine that allows you to execute standard SQL statements to perform interactive analytic queries of big data. For more information, see Overview. DataWorks provides E-MapReduce (EMR) Presto nodes that you can use to develop and periodically schedule Presto tasks. This topic describes the procedure of using an EMR Presto node to develop tasks and the related precautions.
Prerequisites
An Alibaba Cloud EMR cluster is created and registered to DataWorks. For more information, see Register an EMR cluster to DataWorks.
(Required if you use a RAM user to develop tasks) The RAM user is added to the DataWorks workspace as a member and is assigned the Develop or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.
A serverless resource group is purchased and configured. The configurations include association with a workspace and network configuration. For more information, see Create and use a serverless resource group.
A workflow is created in DataStudio.
Development operations in different types of compute engines are performed based on workflows in DataStudio. Therefore, before you create a node, you must create a workflow. For more information, see Create a workflow.
Limits
Only EMR Hadoop clusters support EMR Presto nodes for task development. EMR DataLake clusters and custom clusters do not support EMR Presto nodes.
This type of node can be run only on a serverless resource group or an exclusive resource group for scheduling. We recommend that you use a serverless resource group.
The size of SQL statements for Presto task development cannot exceed 130 KB.
If you use an EMR Presto node to query data, a maximum of 10,000 data records can be returned, and the total size of the returned data records cannot exceed 10 MB.
Data lineage: Tasks that are developed by using EMR Presto nodes do not support generation of data lineages.
Step 1: Create an EMR Presto node
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. Then, choose in the left-side navigation pane. On the page that appears, select the desired workspace from the drop-down list and click Go to DataStudio.
Create an EMR Presto node.
Find the desired workflow, right-click the name of the workflow, and then choose .
NoteAlternatively, you can move the pointer over the Create icon and choose .
In the Create Node dialog box, configure the Name, Engine Instance, Node Type, and Path parameters. Click Confirm. The configuration tab of the EMR Presto node appears.
NoteThe node name can contain only letters, digits, underscores (_), and periods (.).
Step 2: Develop an EMR Presto task
You can develop a Presto task on the configuration tab of the EMR Presto node.
Develop SQL code
In the SQL editor, develop node code. You can define variables in the ${Variable} format in the node code and configure the scheduling parameters that are assigned to the variables as values in the Scheduling Parameter section of the Properties tab. This way, the values of the scheduling parameters are dynamically replaced in the node code when the node is scheduled to run. For more information about how to use scheduling parameters, see Supported formats of scheduling parameters. Sample code:
select '${var}'; -- You can assign a specific scheduling parameter to the var variable.
select * from userinfo ;The size of SQL statements for Presto task development cannot exceed 130 KB.
If multiple EMR data sources are associated with DataStudio in your workspace, you must select one from the data sources based on your business requirements. If only one EMR compute engine is associated with the current workspace, you do not need to select a compute engine.
If you want to change the scheduling parameter that is assigned to the variable in the code, click Run with Parameters in the top toolbar. For information about value assignment for the scheduling parameters, see What are the differences in the value assignment logic of scheduling parameters among the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?
(Optional) Configure advanced parameters
You can configure advanced parameters on the Advanced Settings tab of the configuration tab of the current node. For more information about how to configure the parameters, see Spark Configuration. The following table describes the advanced parameters that can be configured.
Hadoop cluster: created on the EMR on ECS page
Advanced parameter | Description |
FLOW_SKIP_SQL_ANALYZE | The execution method of SQL statements. Valid values:
Note This parameter is available only for testing in the data development environment of a DataWorks workspace. |
USE_GATEWAY | Specifies whether to use a gateway cluster to commit jobs on the current node. Valid values:
Note If the EMR cluster to which the node belongs is not associated with a gateway cluster but the USE_GATEWAY parameter is set to |
Run the Presto task
In the toolbar, click the
icon. In the Parameters dialog box, select the desired resource group from the Resource Group Name drop-down list and click Run. NoteIf you want to access a data source over the Internet or a virtual private cloud (VPC), you must use the resource group for scheduling that is connected to the data source. For more information, see Network connectivity solutions.
If you want to change the resource group in subsequent operations, you can click the
icon to change the resource group in the Parameters dialog box.
Click the
icon in the top toolbar to save the SQL statements. Optional. Perform smoke testing.
You can perform smoke testing on the node in the development environment when you commit the node or after you commit the node. For more information, see Perform smoke testing.
Step 3: Configure scheduling properties
If you want the system to periodically run a task on the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure task scheduling properties based on your business requirements. For more information, see Overview.
You must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
Step 4: Deploy the task
After a task on a node is configured, you must commit and deploy the task. After you commit and deploy the task, the system runs the task on a regular basis based on scheduling configurations.
Click the
icon in the top toolbar to save the task. Click the
icon in the top toolbar to commit the task. In the Submit dialog box, configure the Change description parameter. Then, determine whether to review task code after you commit the task based on your business requirements.
NoteYou must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
You can use the code review feature to ensure the code quality of tasks and prevent task execution errors caused by invalid task code. If you enable the code review feature, the task code that is committed can be deployed only after the task code passes the code review. For more information, see Code review.
If you use a workspace in standard mode, you must deploy the task in the production environment after you commit the task. To deploy a task on a node, click Deploy in the upper-right corner of the configuration tab of the node. For more information, see Deploy nodes.
What to do next
After you commit and deploy the task, the task is periodically run based on the scheduling configurations. You can click Operation Center in the upper-right corner of the configuration tab of the corresponding node to go to Operation Center and view the scheduling status of the task. For more information, see View and manage auto triggered nodes.