This topic describes how to create an E-MapReduce (EMR) Hive node. EMR Hive nodes allow you to use SQL-like statements to read data from and write data to data warehouses with large datasets and manage the data warehouses. The data warehouses are stored in a distributed storage system. You can use EMR Hive nodes to analyze and develop large amounts of log data in an efficient manner.
Prerequisites
An Alibaba Cloud EMR cluster is created and registered to DataWorks. For more information, see Register an EMR cluster to DataWorks.
(Required if you use a RAM user to develop tasks) The RAM user is added to the DataWorks workspace as a member and is assigned the Develop or Workspace Administrator role. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member, see Add workspace members and assign roles to them.
A serverless resource group is purchased and configured. The configurations include association with a workspace and network configuration. For more information, see Create and use a serverless resource group.
A workflow is created in DataStudio.
Development operations in different types of compute engines are performed based on workflows in DataStudio. Therefore, before you create a node, you must create a workflow. For more information, see Create a workflow.
Limits
This type of node can be run only on a serverless resource group or an exclusive resource group for scheduling. We recommend that you use a serverless resource group.
If you want to manage metadata for a DataLake or custom cluster in DataWorks, you must configure EMR-HOOK in your cluster first. If you do not configure EMR-HOOK in your cluster, metadata cannot be displayed in real time, audit logs cannot be generated, and data lineages cannot be displayed in DataWorks. EMR governance tasks also cannot be run. For information about how to configure EMR-HOOK, see Use the Hive extension feature to record data lineage and historical access information.
Step 1: Create an EMR Hive node
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. Then, choose in the left-side navigation pane. On the page that appears, select the desired workspace from the drop-down list and click Go to DataStudio.
Create an EMR Hive node.
Find the desired workflow, right-click the name of the workflow, and then choose .
NoteAlternatively, you can move the pointer over the Create icon and choose .
In the Create Node dialog box, configure the Name, Engine Instance, Node Type, and Path parameters. Click Confirm. The configuration tab of the EMR Hive node appears.
NoteThe node name can contain only letters, digits, underscores (_), and periods (.).
Step 2: Develop an EMR Hive task
You can develop a Hive task on the configuration tab of the EMR Hive node.
Develop SQL code
In the SQL editor, develop node code. You can define variables in the ${Variable} format in the node code and configure the scheduling parameters that are assigned to the variables as values in the Scheduling Parameter section of the Properties tab. This way, the values of the scheduling parameters are dynamically replaced in the node code when the node is scheduled to run. For more information about how to use scheduling parameters, see Supported formats of scheduling parameters. Sample code:
show tables;
select '${var}'; -- You can assign a specific scheduling parameter to the var variable.
select * from userinfo ;The size of the SQL statements for Hive task development cannot exceed 130 KB.
If multiple EMR data sources are associated with DataStudio in your workspace, you must select one from the data sources based on your business requirements. If only one EMR data source is associated with DataStudio in your workspace, you do not need to select a data source.
If you want to change the scheduling parameter that is assigned to the variable in the code, click Run with Parameters in the top toolbar. For more information about the value assignment logic of scheduling parameters, see What are the differences in the value assignment logic of scheduling parameters among the Run, Run with Parameters, and Perform Smoke Testing in Development Environment modes?
(Optional) Configure advanced parameters
You can configure advanced parameters on the Advanced Settings tab of the configuration tab of the current node. For more information about how to configure the parameters, see Spark Configuration. The following table describes the advanced parameters that can be configured for different types of EMR clusters.
DataLake cluster or custom cluster: created on the EMR on ECS page
Advanced parameter | Description |
queue | The scheduling queue to which jobs are committed. Default value: default. For information about EMR YARN, see YARN schedulers. |
priority | The priority. Default value: 1. |
FLOW_SKIP_SQL_ANALYZE | The execution method of SQL statements. Valid values:
Note This parameter is available only for testing in the development environment of a DataWorks workspace. |
DATAWORKS_SESSION_DISABLE | Specifies whether to establish a JDBC connection every time an SQL statement is executed. This parameter is available for testing in the development environment of a DataWorks workspace. Valid values:
Note If the DATAWORKS_SESSION_DISABLE parameter is set to |
Others | You can also add a Hive connection parameter on the Advanced Settings tab for the EMR Hive node. |
Hadoop cluster: created on the EMR on ECS page
Advanced parameter | Description |
queue | The scheduling queue to which jobs are committed. Default value: default. For information about EMR YARN, see YARN schedulers. |
priority | The priority. Default value: 1. |
FLOW_SKIP_SQL_ANALYZE | The execution method of SQL statements. Valid values:
Note This parameter is available only for testing in the development environment of a DataWorks workspace. |
USE_GATEWAY | Specifies whether to use a gateway cluster to commit jobs on the current node. Valid values:
Note If the EMR cluster to which the node belongs is not associated with a gateway cluster but the USE_GATEWAY parameter is set to |
Run the Hive task
In the toolbar, click the
icon. In the Parameters dialog box, select the desired resource group from the Resource Group Name drop-down list and click Run. NoteIf you want to access a data source over the Internet or a virtual private cloud (VPC), you must use the resource group for scheduling that is connected to the data source. For more information, see Network connectivity solutions.
If you want to change the resource group in subsequent operations, you can click the
(Run with Parameters) icon to change the resource group in the Parameters dialog box. If you use an EMR Hive node to query data, a maximum of 10,000 data records can be returned, and the total size of the returned data records cannot exceed 10 MB.
Click the
icon in the top toolbar to save the SQL statements. Optional. Perform smoke testing.
You can perform smoke testing on the node in the development environment when you commit the node or after you commit the node. For more information, see Perform smoke testing.
Step 3: Configure scheduling properties
If you want the system to periodically run a task on the node, you can click Properties in the right-side navigation pane on the configuration tab of the node to configure task scheduling properties based on your business requirements. For more information, see Overview.
You must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
Step 4: Deploy the task
After a task on a node is configured, you must commit and deploy the task. After you commit and deploy the task, the system runs the task on a regular basis based on scheduling configurations.
Click the
icon in the top toolbar to save the task. Click the
icon in the top toolbar to commit the task. In the Submit dialog box, configure the Change description parameter. Then, determine whether to review task code after you commit the task based on your business requirements.
NoteYou must configure the Rerun and Parent Nodes parameters on the Properties tab before you commit the task.
You can use the code review feature to ensure the code quality of tasks and prevent task execution errors caused by invalid task code. If you enable the code review feature, the task code that is committed can be deployed only after the task code passes the code review. For more information, see Code review.
If you use a workspace in standard mode, you must deploy the task in the production environment after you commit the task. To deploy a task on a node, click Deploy in the upper-right corner of the configuration tab of the node. For more information, see Deploy nodes.
What to do next
After you commit and deploy the task, the task is periodically run based on the scheduling configurations. You can click Operation Center in the upper-right corner of the configuration tab of the corresponding node to go to Operation Center and view the scheduling status of the task. For more information, see View and manage auto triggered nodes.