DataWorks:Usage notes for development of CDP or CDH tasks in DataWorks

Fee

Description

Fees for the DataWorks edition that you use

You must activate DataWorks before you can develop tasks in DataWorks. If you activate DataWorks Standard Edition, DataWorks Professional Edition, or DataWorks Enterprise Edition, you are charged the fees for the edition when you purchase the edition.

Fees for the scheduling resources that you use to schedule tasks

After tasks are developed, scheduling resources are required to schedule the tasks. You can purchase resource groups for scheduling, such as subscription exclusive resource groups for scheduling and the pay-as-you-go shared resource group for scheduling, based on your business requirements, and pay for the resource groups.

Fees for the resources that you use to synchronize data

A data synchronization task consumes scheduling resources and synchronization resources. You can purchase resource groups for Data Integration, such as subscription exclusive resource groups for Data Integration and the pay-as-you-go shared resource group for Data Integration (debugging), based on your business requirements, and pay for the resource groups.

Fee

Description

Database fees

When you run data synchronization tasks to read data from and write data to databases, database fees may be generated.

Computing and storage fees

When you run tasks of a specific type of compute engine, computing and storage fees of this type of compute engine may be generated.

Network service fees

When you establish network connections between DataWorks and other related services, network service fees may be generated. For example, if you use services, such as Express Connect, Elastic IP Address (EIP), and Internet Shared Bandwidth, to establish network connections between DataWorks and other related services, you may be charged network service fees.

Item

Description

References

Select a DataWorks edition

DataWorks Basic Edition allows you to perform the following basic operations during the development of CDP or CDH data: migrate data to the cloud, develop data, schedule tasks, and govern data. If you want to use more advanced data governance and data security solutions, you can purchase DataWorks of an advanced edition, such as DataWorks Standard Edition, DataWorks Professional Edition, or DataWorks Enterprise Edition.

Differences among DataWorks editions

Select a resource group

You can use a serverless resource group or an old-version exclusive resource group for scheduling to run tasks in a CDP or CDH cluster. We recommend that you use a serverless resource group.

Create and use a serverless resource group

Item

Description

References

Prepare a data synchronization environment

Before you develop a data synchronization task in DataWorks based on the Hive service that is deployed in a cluster, you must add the Hive service to a DataWorks workspace as a data source.

Supported data source types and synchronization operations

Prepare an environment for data development and analysis

Before you use DataWorks to periodically schedule CDP or CDH tasks, you must add a CDP or CDH cluster to DataWorks as a data source. Then, you can use the data source to perform operations, such as data development, data analysis, and periodic task scheduling.

Register a CDH or CDP cluster to DataWorks

Prepare a collaborative development environment

To ensure that RAM users can collaborate with each other to develop data in a workspace, you must perform the following operations:

• Add the RAM users to the current workspace as members and assign the Development role to the RAM users in the workspace.

• Add the workspace members to the desired CDP or CDH cluster.

Add a RAM user to a workspace as a member and assign roles to the member

Module

Description

References

Data Modeling

Data Modeling is the first step for end-to-end data governance. Data Modeling uses the modeling methodology of the Alibaba data mid-end, interprets the business data of an enterprise from a business perspective by using the data warehouse planning, data standard, dimensional modeling, and data metric modules, and allows personnel inside the enterprise to quickly understand and share the idea of measuring and interpreting business data in compliance with data warehousing specifications.

Data Modeling overview

DataStudio

DataWorks encapsulates the capabilities of a CDP or CDH compute engine. This way, you can use the CDP or CDH compute engine to run CDP or CDH data synchronization and development tasks.

• Data synchronization: DataStudio supports only specific batch and real-time synchronization scenarios. For more information about data synchronization scenarios, see Data Integration overview.

• Data development: You can develop and periodically schedule different types of tasks in DataWorks without the need to use complex command lines.

• Create a CDH Hive node

• Create a CDH Spark node

• Create a CDH MR node

• Create a CDH Presto node

• Create a CDH Impala node

You can use general nodes and nodes of a specific type of compute engine in DataWorks to process complex logic.

DataWorks supports the following types of general nodes:

• Zero load nodes that are used to manage workflows

• HTTP Trigger nodes that are used in the scenarios in which external scheduling systems are used to trigger scheduling of nodes in DataWorks, OSS object inspection nodes, and FTP Check nodes

• Assignment nodes that are used to pass input parameters and output parameters for nodes, and parameter nodes

• Do-while nodes that are used to execute node code in loops, for-each nodes that are used to traverse the outputs of assignment nodes in loops and judge the outputs, and branch nodes

• Other nodes, such as common Shell nodes and MySQL database nodes

• Create and use a zero load node

• Create an HTTP Trigger node

• OSS object inspection node

• Create an FTP Check node

• Configure an assignment node

• Create a parameter node

• Logic of do-while nodes

• Logic of for-each nodes

• Branch node

After tasks on nodes are developed, you can perform the following operations based on your business requirements:

• Configure scheduling properties for nodes

  If you want to enable DataWorks to periodically run your tasks on nodes, you must configure scheduling properties for the nodes, such as scheduling dependencies and scheduling parameters.

• Debug nodes

  To ensure that tasks on nodes in the production environment are run in an efficient manner and prevent a waste of computing resources, we recommend that you debug and run the tasks before you deploy the tasks.

• Deploy nodes

  The tasks on nodes can be scheduled to run only after they are deployed to the production environment. Therefore, after the tasks are developed, you must deploy the tasks to the production environment. After the tasks are deployed, you can view and manage the tasks on the Cycle Task page in Operation Center.

• Manage nodes

  You can perform various operations on the tasks on nodes, such as deploying and undeploying the tasks, and modifying scheduling properties for multiple tasks at the same time.

• Perform process management

  DataWorks provides process control for task development and deployment to ensure the accuracy and security of the operations that are performed on tasks. For example, DataWorks provides the code review, forceful smoke testing, and code review logic customization features.

• Overview

• Debugging procedure

• Deploy nodes

• Perform operations on multiple DataWorks objects at a time

• Process management

Operation Center

Operation Center is an end-to-end big data O&M and monitoring platform. Operation Center allows you to view the status of tasks and perform O&M operations on tasks on which exceptions occur. For example, you can perform intelligent diagnostics and rerun tasks in Operation Center. Operation Center provides the intelligent baseline feature that you can use to resolve issues such as uncontrollable output time of important tasks and difficulties in monitoring of massive tasks. This feature helps you ensure the timeliness of task output.

Perform basic O&M operations on auto triggered nodes

Data Quality

Data Quality ensures data availability for the end-to-end data R&D process and provides reliable data for your business in an efficient manner. Data Quality can help you identify data quality issues at the earliest opportunity and prevent data quality issues from escalating by virtue of effective monitoring rule-based quality checks and the combination of monitoring rules and task scheduling processes.

Data Quality overview

Module

Description

References

Data Map

Data Map is an enterprise-grade data management platform that provides management, sorting, lineage viewing, quick search, and in-depth understanding capabilities for data objects based on the underlying unified metadata services.

Data Map overview

Security Center

Data Security Guard

Approval Center

The Security Center, Data Security Guard, and Approval Center modules serve as an end-to-end data security governance platform that provides the following features: data categorization and sensitivity level classification, sensitive data identification, management on data-related authorization, masking of sensitive data, audit of access to sensitive data, and risk identification and response. This helps you implement data security governance.

• Security Center overview

• Data Security Guard overview

• Approval Center overview

Data Governance Center

Data Governance Center automatically identifies items to be governed for multiple governance fields based on rules that come from experience in data-related fields, and provides governance and optimization solutions covering pre-event issue prevention and post-event issue resolution. Data Governance Center can help you actively and systematically complete data governance.

Data Governance Center overview

Item

Description

References

OpenAPI

The OpenAPI module allows you to call DataWorks API operations so that you can integrate your applications with DataWorks. This can help facilitate big data processing, decrease manual operations and O&M operations, minimize data risks, and reduce costs for enterprises.

OpenAPI

OpenEvent

The OpenEvent module allows you to subscribe to DataWorks change events related to your applications so that you can detect and respond to the changes at the earliest opportunity.

OpenEvent overview

Extensions

You can use the OpenEvent module to subscribe to event messages that are generated in your DataWorks workspace. You can use the Extensions module to register your local program as an extension to manage extension point events and processes.

Extensions overview