DataWorks offers various MaxCompute node types for developing different tasks and provides a rich set of scheduling configurations for flexible task setup. This topic describes how to create and manage MaxCompute nodes.
Usage notes
The account used to develop MaxCompute tasks must be added to the corresponding workspace and have the Development or Workspace Manager role. Grant the Workspace Manager role with caution, as it includes extensive permissions. For more information about how to add members, see Add members to a workspace.
Create a MaxCompute node
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Associate a MaxCompute compute resource with the workspace and create a workflow.
Data Studio uses workflows to manage development for different engines, so you must create a workflow before creating a node.
This section demonstrates how to create a node by using an ODPS SQL node as an example.
Right-click a workflow and choose . You can also click Create in the top menu bar and follow the on-screen instructions to create a MaxCompute node.
ImportantIf the option is not available, click Computing Resource in the navigation pane to check whether a MaxCompute compute resource is associated. You must associate a MaxCompute compute resource and then refresh the page before you can create nodes.
In the dialog box that appears, enter a name for the node and click Confirm. After the node is created, you can develop and configure the MaxCompute task in the node editor.
Develop MaxCompute tasks
DataWorks supports multiple MaxCompute node types for developing various MaxCompute tasks.
When you run a MaxCompute task, a cost estimate is displayed. This fee, charged by MaxCompute, is only an estimate. The actual cost is reflected on your bill. For more information about billing, see Billable items and billing methods.
An error might occur if the table does not exist or if you lack the required permissions. You can ignore this error for now and handle it based on the specific error message after the node runs.
Node type | Use case | Development guide |
ODPS SQL | Develop MaxCompute SQL tasks. | |
SQL Snippet | Develop MaxCompute SQL tasks. When you have multiple SQL procedures with similar logic but different table names, you can abstract that logic into an SQL snippet. Define variable input and output tables as parameters to enable code reuse. | |
PyODPS 3 | Develop MaxCompute PyODPS tasks. The PyODPS 3 node uses Python 3. | |
PyODPS 2 | Develop MaxCompute PyODPS tasks. The PyODPS 2 node uses Python 2. | |
ODPS Spark | Develop MaxCompute Spark tasks. | |
ODPS Script | Develop MaxCompute SQL script tasks. | |
ODPS MR | Develop MaxCompute MapReduce tasks. |
Create tables, resources, and functions
DataWorks also provides tools for managing tables, resources, and functions to help you streamline task development.
MaxCompute tables: You can use the DataWorks UI to create tables, view table information, and manage tables. For more information, see Create and use MaxCompute tables and Manage tables.
MaxCompute functions and resources:
When you develop MaxCompute tasks in DataWorks, you can directly use MaxCompute's built-in functions. For a list of built-in functions, see Use built-in functions.
You can create a UDF as a MaxCompute resource in DataWorks and register it as a MaxCompute function. You can then call the function directly during task development. For more information, see Create and use MaxCompute resources and Create and use user-defined functions.
You can upload a locally developed resource package to DataWorks or create a resource directly in DataWorks.
DataWorks allows you to upload text files, Python code, and compressed packages such as .zip, .tgz, .tar.gz, .tar, and .jar files as various types of resources to MaxCompute. These resources can be read and used in UDFs and MapReduce jobs. For more information about how to upload and use resources, see Create and use MaxCompute resources.
Next steps
After developing the task, you can proceed with the following steps:
Scheduling: Configure the scheduling properties for the node. If the task must run periodically, you must configure properties such as rerun settings and scheduling dependencies. For more information, see Overview of task scheduling properties.
Task debugging: Test and run the code of the current node to verify that the code logic is correct. For more information, see Task debugging process.
Task deployment: After you complete all development-related operations, you must deploy all nodes. After deployment, the nodes run periodically based on their scheduling configurations. For more information, see Deploy tasks.