DataWorks:Usage notes for development of MaxCompute tasks in DataWorks

Fee

Description

Fees for the DataWorks edition that you use

You must activate DataWorks before you can develop tasks in DataWorks. If you activate DataWorks Standard Edition, DataWorks Professional Edition, or DataWorks Enterprise Edition, you are charged the fees for the edition when you purchase the edition.

Fees for the scheduling resources that you use to schedule tasks

After tasks are developed, scheduling resources are required to schedule the tasks. You can purchase a serverless resource group or an old-version exclusive resource group for scheduling, and pay for the resource group. We recommend that you purchase a serverless resource group.

Fees for the resources that you use to synchronize data

A data synchronization task consumes scheduling resources and synchronization resources. You can purchase a serverless resource group or an old-version exclusive resource group for Data Integration, and pay for the resource group. We recommend that you purchase a serverless resource group.

Fee

Description

Database fees

When you run data synchronization tasks to read data from and write data to databases, database fees may be generated.

Computing and storage fees

When you run tasks of a specific type of compute engine, computing and storage fees of this type of compute engine may be generated. For example, if you run a task on an ODPS SQL node to create a MaxCompute table and write data to the MaxCompute table, you may be charged for computing and storage resources of a MaxCompute compute engine.

Network service fees

When you establish network connections between DataWorks and other related services, network service fees may be generated. For example, if you use services, such as Express Connect, Elastic IP Address (EIP), and Internet Shared Bandwidth, to establish network connections between DataWorks and other related services, you may be charged network service fees.

Item

Description

References

Select a DataWorks edition

DataWorks Basic Edition allows you to perform the following basic operations during the development of MaxCompute data: migrate data to the cloud, develop data, schedule MaxCompute tasks, and govern data. If you want to use more advanced data governance and data security solutions, you can purchase DataWorks of an advanced edition, such as DataWorks Standard Edition, DataWorks Professional Edition, or DataWorks Enterprise Edition.

Comparison between DataWorks Basic Edition and DataWorks advanced editions and edition upgrade description

Select a resource group

• Serverless resource group (recommended): You can use one serverless resource group in data synchronization, scheduling, and DataService Studio at the same time. You can plan and allocate resources based on your business requirements.

• Old-version resource group (exclusive resource group or shared resource group): Basic scheduling requirements can be met. Old-version resource groups will be discontinued in the future.

Overview

Item

Description

References

Prepare a data synchronization environment

Before you develop a MaxCompute data synchronization task in DataWorks, you must add a MaxCompute project to a DataWorks workspace as a data source.

Add a MaxCompute data source

Prepare an environment for data development and analysis

Before you schedule MaxCompute tasks in DataWorks, you must add a MaxCompute project to a DataWorks workspace as a data source and associate the data source with DataStudio. Then, you can use the data source to perform operations, such as data development, data analysis, and periodic task scheduling.

• Add a MaxCompute data source

• Preparations before data development: Associate a data source or a cluster with DataStudio

Prepare a collaborative development environment

To ensure that RAM users can collaborate with each other to develop data in a workspace, you must add the RAM users to the workspace as members and assign the Development role to the RAM users in the workspace.

Add a RAM user to a workspace as a member and assign roles to the member

Service

Description

References

Data Modeling

Data Modeling is the first step for end-to-end data governance. Data Modeling uses the modeling methodology of the Alibaba data mid-end, interprets the business data of an enterprise from a business perspective by using the data warehouse planning, data standard, dimensional modeling, and data metric modules, and allows personnel inside the enterprise to quickly understand and share the idea of measuring and interpreting business data in compliance with data warehousing specifications.

Data Modeling overview

DataStudio

DataWorks encapsulates the capabilities of a MaxCompute compute engine. This way, you can use the MaxCompute compute engine to run MaxCompute data synchronization and development tasks.

• Data synchronization: You can synchronize data from another type of data source to a MaxCompute data source or synchronize data from a MaxCompute data source to another type of data source. DataStudio supports only some of the batch synchronization and real-time synchronization scenarios. For more information about data synchronization scenarios, see Data Integration.

• Data development: You can develop and periodically schedule different types of tasks in DataWorks without the need to use complex command lines.

• Definition of the code of a node: MaxCompute syntax

• Create and manage MaxCompute tables

• Develop a MaxCompute SQL task

• Reference a script template

• Develop a MaxCompute Spark task

• Develop a PyODPS 2 task

• Develop a PyODPS 3 task

• Develop a MaxCompute script task

• Develop a MaxCompute MR task

• Create and use MaxCompute resources

• Create and use a MaxCompute UDF

You can use general nodes and nodes of a specific type of compute engine in DataWorks to process complex logic.

DataWorks supports the following types of general nodes:

• Zero load nodes that are used to manage workflows

• HTTP Trigger nodes that are used in the scenarios in which external scheduling systems are used to trigger scheduling of nodes in DataWorks, OSS object inspection nodes, and FTP Check nodes

• Assignment nodes that are used to pass input parameters and output parameters for nodes, and parameter nodes

• Do-while nodes that are used to execute node code in loops, for-each nodes that are used to traverse the outputs of assignment nodes in loops and judge the outputs, and branch nodes

• Other nodes, such as common Shell nodes and MySQL database nodes

• Create and use a zero load node

• Create an HTTP Trigger node

• OSS object inspection node

• Create an FTP Check node

• Configure an assignment node

• Create a parameter node

• Logic of do-while nodes

• Logic of for-each nodes

• Branch node

After tasks on nodes are developed, you can perform the following operations based on your business requirements:

• Configure scheduling properties for nodes

  If you want to enable DataWorks to periodically run your tasks on nodes, you must configure scheduling properties for the nodes, such as scheduling dependencies and scheduling parameters.

• Debug nodes

  To ensure that tasks on nodes in the production environment are run in an efficient manner and prevent a waste of computing resources, we recommend that you debug and run the tasks before you deploy the tasks.

• Deploy nodes

  The tasks on nodes can be scheduled to run only after they are deployed to the production environment. Therefore, after the tasks are developed, you must deploy the tasks to the production environment. After the tasks are deployed, you can view and manage the tasks on the Cycle Task page in Operation Center.

• Manage nodes

  You can perform various operations on the tasks on nodes, such as deploying and undeploying the tasks, and modifying scheduling properties for multiple tasks at the same time.

• Perform process management

  DataWorks provides process control for task development and deployment to ensure the accuracy and security of the operations that are performed on tasks. For example, DataWorks provides the code review, forceful smoke testing, and code review logic customization features.

• Overview

• Debugging procedure

• Deploy nodes

• Perform operations on multiple DataWorks objects at a time

• Process management

Operation Center

Operation Center is an end-to-end big data O&M and monitoring platform. Operation Center allows you to view the status of tasks and perform O&M operations on tasks on which exceptions occur. For example, you can perform intelligent diagnostics and rerun tasks in Operation Center. Operation Center provides the intelligent baseline feature that you can use to resolve issues such as uncontrollable output time of important tasks and difficulties in monitoring of massive tasks. This feature helps you ensure the timeliness of task output.

Perform basic O&M operations on auto triggered nodes

Data Quality

Data Quality ensures data availability for the end-to-end data R&D process and provides reliable data for your business in an efficient manner. Data Quality can help you identify data quality issues at the earliest opportunity and prevent data quality issues from escalating by virtue of effective monitoring rule-based quality checks and the combination of monitoring rules and task scheduling processes.

Data Quality overview

Module

Description

References

Data Map

Data Map is an enterprise-grade data management platform that provides management, sorting, quick search, and in-depth understanding capabilities for data objects based on the underlying unified metadata services.

Data Map overview

Security Center

Data Security Guard

Approval Center

Security Center is an end-to-end data security governance platform that covers classification of data assets, sensitive data identification, management on data-related authorization, masking of sensitive data, audit of access to sensitive data, and risk identification and response. Security Center helps you determine data security governance issues.

• Security Center overview

• Data Security Guard overview

• Approval Center overview

Data Governance Center

Data Governance Center automatically identifies items to be governed for multiple governance fields based on rules that come from experience in data-related fields, and provides governance and optimization solutions covering pre-event issue prevention and post-event issue resolution. Data Governance Center can help you actively and systematically complete data governance.

Data Governance Center overview

Item

Description

References

OpenAPI

The OpenAPI module allows you to call DataWorks API operations so that you can integrate your applications with DataWorks. This can help facilitate big data processing, decrease manual operations and O&M operations, minimize data risks, and reduce costs for enterprises.

OpenAPI

OpenEvent

The OpenEvent module allows you to subscribe to DataWorks change events related to your applications so that you can detect and respond to the changes at the earliest opportunity.

OpenEvent overview

Extensions

You can use the OpenEvent module to subscribe to event messages that are generated in your DataWorks workspace. You can use the Extensions module to register your local program as an extension to manage extension point events and processes.

Extensions overview