DataWorks:Usage notes for development of Hologres tasks in DataWorks

Fee

Description

Fees for the DataWorks edition that you use

You must activate DataWorks before you can develop tasks in DataWorks. If you activate DataWorks Standard Edition, DataWorks Professional Edition, or DataWorks Enterprise Edition, you are charged the fees for the edition when you purchase the edition.

Fees for the scheduling resources that you use to schedule tasks

After tasks are developed, scheduling resources are required to schedule the tasks. You can purchase a serverless resource group or an old-version exclusive resource group for scheduling, and pay for the resource group. We recommend that you purchase a serverless resource group.

Fees for the resources that you use to synchronize data

A data synchronization task consumes scheduling resources and synchronization resources. You can purchase a serverless resource group or an old-version exclusive resource group for Data Integration, and pay for the resource group. We recommend that you purchase a serverless resource group.

Fee

Description

Database fees

When you run data synchronization tasks to read data from and write data to databases, database fees may be generated.

Computing and storage fees

When you run tasks of a specific compute engine type, computing and storage fees of this type of compute engine may be generated. For example, if you create and run a Hologres SQL task to query Hologres data, you may be charged for computing and storage resources of a Hologres compute engine.

Network service fees

When you establish network connections between DataWorks and other related services, network service fees may be generated. For example, if you use services, such as Express Connect, Elastic IP Address (EIP), and Internet Shared Bandwidth, to establish network connections between DataWorks and other related services, you may be charged network service fees.

Item

Description

References

Select a DataWorks edition

DataWorks Basic Edition allows you to perform the following basic operations during the development of Hologres data: migrate data to the cloud, develop data, schedule Hologres tasks, and govern data. If you want to use more advanced data governance and data security solutions, you can purchase DataWorks Standard Edition, DataWorks Professional Edition, or DataWorks Enterprise Edition.

Comparison among DataWorks Standard Edition, DataWorks Professional Edition, and DataWorks Enterprise Edition and their upgrade descriptions

Select a resource group

• Serverless resource group (recommended): You can use one serverless resource group in data synchronization, scheduling, and DataService Studio at the same time. You can plan and allocate resources based on your business requirements.

• Old-version resource group (exclusive resource group or shared resource group): Basic scheduling requirements can be met. Old-version resource groups will be discontinued in the future.

Overview

Item

Description

References

Prepare a data synchronization environment

Before you run Hologres synchronization tasks in DataWorks to synchronize data from or to Hologres, you must add a Hologres instance to a DataWorks workspace as a data source. You can configure the synchronization tasks for the data source only after the data source is added.

Add a Hologres data source

Prepare an environment for data development and analysis

Before you use DataWorks to schedule a Hologres task, you must add a Hologres instance to a DataWorks workspace as a data source and associate the data source with DataStudio. Then, you can perform operations, such as data development, data analysis, and periodic task scheduling, based on the data source.

• Add a Hologres data source

• Preparations before data development: Associate a data source or a cluster with DataStudio

Prepare a collaborative development environment

To ensure that RAM users can collaborate with each other to develop data in a workspace, you must perform the following operations:

• Add the RAM users to the workspace as members and assign the Development role to the RAM users in the workspace.

• Add workspace members to a Hologres compute engine instance and to a Hologres data source that you associate with a DataWorks workspace, and grant the required permissions on databases to a RAM user that you use to run tasks in the production environment.

• Add members to a workspace

• Configure permissions on the Hologres compute engine for a workspace member

Module

Description

References

Data Modeling

Data Modeling is the first step for end-to-end data governance. Data Modeling uses the modeling methodology of the Alibaba data mid-end, interprets the business data of an enterprise from a business perspective by using the data warehouse planning, data standard, dimensional modeling, and data metric modules, and allows personnel inside the enterprise to quickly understand and share the idea of measuring and interpreting business data in compliance with data warehousing specifications.

Data Modeling overview

DataStudio

DataWorks encapsulates capabilities of a Hologres compute engine and allows you to run Hologres data synchronization tasks and Hologres data development tasks.

• Data synchronization: You can synchronize data between a Hologres data source and another type of data source. DataStudio supports only specific batch synchronization and real-time synchronization scenarios. For more information about use scenarios of data synchronization, access the Data Integration module.

• Data development: You can develop and allow the system to periodically schedule different types of tasks in DataWorks without the need to use complex command lines.

• For information about the definition of the code of a node, see Hologres development standards.

• Create a Hologres SQL node

• Create a node to synchronize schemas of MaxCompute tables with a few clicks

• Create a node to synchronize MaxCompute data with a few clicks

You can use general nodes and nodes of a specific type of compute engine in DataWorks to process complex logic.

DataWorks supports the following types of general nodes:

• Zero load nodes that are used to manage workflows

• HTTP Trigger nodes that are used in the scenarios in which external scheduling systems are used to trigger scheduling of nodes in DataWorks, OSS object inspection nodes, and FTP Check nodes

• Assignment nodes that are used to pass input parameters and output parameters for nodes, and parameter nodes

• Do-while nodes that are used to execute node code in loops, for-each nodes that are used to traverse the outputs of assignment nodes in loops and judge the outputs, and branch nodes

• Other nodes, such as common Shell nodes and MySQL database nodes

• Shell node

• Zero load node

• Assignment node

• Parameter node

• For-each node

• Do-while node

• Merge node

• Branch node

• Check node

• SSH node

• FTP Check node

• HTTP Trigger node

• Data Lake Analytics node

• Function Compute node

• Data quality monitoring node

After tasks are developed based on nodes, you can perform the following operations based on your business requirements:

• Configure scheduling properties for nodes

  If you want to enable DataWorks to periodically run your tasks on nodes, you must configure scheduling properties for the nodes, such as scheduling dependencies and scheduling parameters.

• Debug nodes

  To ensure that tasks on nodes in the production environment are run in an efficient manner and prevent a waste of computing resources, we recommend that you debug and run the tasks before you deploy the tasks.

• Deploy nodes

  The tasks on nodes can be scheduled to run only after they are deployed to the production environment. Therefore, after the tasks are developed, you must deploy the tasks to the production environment. After the tasks are deployed, you can view and manage the tasks on the Cycle Task page in Operation Center.

• Manage nodes

  You can perform various operations on the tasks on nodes, such as deploying and undeploying the tasks, and modifying scheduling properties for multiple tasks at the same time.

• Perform process management

  DataWorks provides process control for task development and deployment to ensure the accuracy and security of the operations that are performed on tasks. For example, DataWorks provides the code review, forceful smoke testing, and code review logic customization features.

• Overview

• Debugging procedure

• Deploy nodes

• Perform operations on multiple DataWorks objects at a time

• Process management

Operation Center

Operation Center is an end-to-end big data O&M and monitoring platform. Operation Center allows you to view the status of tasks and perform O&M operations on tasks on which exceptions occur. For example, you can perform intelligent diagnostics and rerun tasks in Operation Center. Operation Center provides the intelligent baseline feature that you can use to resolve issues such as uncontrollable output time of important tasks and difficulties in monitoring of massive tasks. This feature helps you ensure the timeliness of task output.

Perform basic O&M operations on auto triggered nodes

Data Quality

Data Quality ensures data availability for the end-to-end data R&D process and provides reliable data for your business in an efficient manner. Data Quality can help you identify data quality issues at the earliest opportunity and prevent data quality issues from escalating by virtue of effective monitoring rule-based quality checks and the combination of monitoring rules and task scheduling processes.

Data Quality overview

Module

Description

References

Data Map

Data Map is an enterprise-grade data management platform that provides management, sorting, quick search, and in-depth understanding capabilities for data objects based on the underlying unified metadata services.

Data Map overview

Security Center

Data Security Guard

Approval Center

Security Center is an end-to-end data security governance platform that covers classification of data assets, sensitive data identification, management on data-related authorization, masking of sensitive data, audit of access to sensitive data, and risk identification and response. Security Center helps you determine data security governance issues.

• Security Center overview

• Data Security Guard overview

• Approval Center overview

Data Governance Center

Data Governance Center automatically identifies items to be governed for multiple governance fields based on rules that come from experience in data-related fields, and provides governance and optimization solutions covering pre-event issue prevention and post-event issue resolution. Data Governance Center can help you actively and systematically complete data governance.

Data Governance Center overview

Item

Description

References

OpenAPI

The OpenAPI module allows you to call DataWorks API operations so that you can integrate your applications with DataWorks. This can help facilitate big data processing, decrease manual operations and O&M operations, minimize data risks, and reduce costs for enterprises.

OpenAPI

OpenEvent

The OpenEvent module allows you to subscribe to DataWorks change events related to your applications so that you can detect and respond to the changes at the earliest opportunity.

OpenEvent overview

Extensions

You can use the OpenEvent module to subscribe to event messages that are generated in your DataWorks workspace. You can use the Extensions module to register your local program as an extension to manage extension point events and processes.

Extensions overview