The DataStudio service of DataWorks allows you to define the development and scheduling properties of auto triggered tasks. DataStudio works with Operation Center to provide a visualized development interface for tasks of various types of compute engines, such as MaxCompute, Hologres, and E-MapReduce (EMR). You can configure settings on the visualized development interface to perform intelligent code development, multi-engine task orchestration in workflows, and standardized task deployment. This way, you can build offline data warehouses, real-time data warehouses, and ad hoc analysis systems to ensure efficient and stable data production. This topic describes the terms that are used in DataStudio, the capabilities provided by DataStudio, and preparations before data development in DataStudio.
Go to the DataStudio page
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
Module introduction
Capability overview
The following figure shows the main features provided by DataStudio. For more information, see the Appendix: Terms related to data development section in this topic.
Feature | Description |
Object organization and management | DataStudio provides a mechanism to organize and manage objects in DataWorks.
For more information, see Create a workflow and the Management modes section in this topic. Note Limits on the maximum numbers of workflows and objects that you can create in DataStudio in each workspace:
If the numbers of workflows and objects in the current workspace reach the upper limit, you can no longer create a workflow or object. |
Task development |
For information about the node types that are supported by DataWorks, see DataWorks nodes. |
Task scheduling |
For more information about task scheduling, see Configure time properties and Scheduling dependency configuration guide. |
Task debugging | You can debug a task or a workflow. For more information, see Debugging procedure. |
Process control | DataStudio provides a standardized task deployment mechanism and various methods to perform process control. You can perform operations that include but are not limited to the following operations for process control:
|
Other features |
|
Introduction to the DataStudio page
You can follow the instructions that are described in Features on the DataStudio page to use the features of each module on the DataStudio page.
Development process
DataWorks DataStudio allows you to create different compute engine types of real-time synchronization tasks, batch synchronization tasks, batch processing tasks, and manually triggered tasks. For more information about data synchronization, see Overview of Data Integration. The configuration requirements on tasks of different compute engine types vary. Take note of the precautions and related instructions on the development of tasks of different compute engine types in DataWorks before you develop tasks based on the task type.
Instructions on the development of tasks of different compute engine types: You can add different data sources to DataWorks to develop tasks in DataWorks. The configuration requirements on tasks of different compute engine types vary. For more information, see the following topics:
Common development process: The following two workspace modes are available: standard mode and basic mode. The node development process varies based on the workspace mode.
Task development process in a workspace in standard mode
Task development process in a workspace in basic mode
Basic process: For example, you want to develop tasks in a workspace in standard mode. The development process includes the following stages: development, debugging, configuration of scheduling settings, task committing, task deployment, and O&M. For more information, see General development process.
Process control: During task development, you can perform operations such as code review and smoke testing provided by DataStudio and use check items preset in Data Governance Center and verification logic customized based on extensions in Open Platform to ensure that specified standards and requirements on task development are met.
NoteThe process control operations vary based on the workspace mode. The actual process control operations shall prevail.
Management modes
A workflow is a basic unit for code development and resource management. A workflow is an abstract business entity that allows you to develop code based on your business requirements. Workflows and nodes in different workspaces are separately developed. For more information about workflows, see Create a workflow.
Workflows can be displayed in a directory tree or in a panel. The display modes enable you to organize code from the business perspective and show the resource classification and business logic in a more efficient manner.
The directory tree allows you to organize your code by task type.
The panel shows the business logic in a workflow.
Get started with DataStudio
Environment preparation
If you want to perform data modeling or data development, or periodically schedule tasks in Operation Center in DataWorks, you must associate your data source or cluster with DataStudio. This way, you can read data in the data source or cluster and perform data development operations.
Add a data source or cluster of a specific type based on the type of tasks that you want to develop and schedule.
Data source or cluster type
Description
The first time you add a MaxCompute data source to DataWorks, DataWorks automatically associates the data source with DataStudio. You do not need to follow the instructions that are described in this topic to manually associate the data source with DataStudio. For MaxCompute data sources that are added later, you must manually associate the data sources with DataStudio.
After you add a data source of one of these types, you must follow the instructions that are described in this topic to manually associate the data source with DataStudio.
After you register a cluster to DataWorks, DataWorks associates the cluster with DataStudio. You do not need to follow the instructions that are described in this topic to manually associate the cluster with DataStudio.
Cloudera's Distribution Including Apache Hadoop (CDH) or Cloudera Data Platform (CDP)
Go to the DataStudio page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
In the left-side navigation pane, click Computing Resource.
If the Computing Resource module is not displayed in the left-side navigation pane, you must go to the Personal Settings tab and select Computing Resource in the DataStudio Modules section to allow the Computing Resource module to be displayed in the left-side navigation pane of the DataStudio page. For more information, see Configure settings in the DataStudio Modules section.
Associate a data source or cluster.
On the Computing Resource page, search for the desired data source or cluster by computing resource name or computing resource type and click Associate. After you associate the data source or cluster with DataStudio, you can read data from the data source or cluster based on the connection information and perform relevant data development operations.
NoteIf data source or cluster information changes, but the data on the current page is not updated in time, refresh the current page to update the cached data.
A data source or a cluster may fail to be associated with DataStudio in the following scenarios:
The configurations of the data sources or clusters of specific types do not support association with DataStudio. For example, you cannot associate a data source that is added by using an AccessKey pair with DataStudio. For more information about limits on the association, see the descriptions that are displayed in the DataWorks console when you associate a data source or a cluster with DataStudio.
The configurations in the development or production environment are missing.
A MaxCompute data source cannot be associated with multiple DataWorks workspaces at the same time.
NoteThe reason why a data source or cluster cannot be associated with DataStudio varies based on the type of the data source or cluster. You can troubleshoot issues based on the reason that is displayed when you try to associate the data source or cluster with DataStudio.
Only the following types of data sources or clusters can be associated with DataStudio: MaxCompute, EMR, Hologres, AnalyticDB for MySQL, ClickHouse, CDH, CDP, and AnalyticDB for PostgreSQL.
The types and number of data sources or clusters that can be associated with DataStudio vary based on the DataWorks edition. For more information, see the Feature comparison section of the "Differences among DataWorks editions" topic.
Quick start tutorial
You can refer to Getting started with data development to learn the basic operations in data development and the data development process.
Node types supported by DataStudio
The DataStudio service of DataWorks allows you to create various types of nodes. You can enable DataWorks to periodically schedule instances that are generated for nodes. You can also select a specific type of node to develop data based on your business requirements. For more information about the node types that are supported by DataWorks, see DataWorks nodes.
Appendix: Terms related to data development
Terms related to task development
Term
Description
Solution
A collection of workflows. A solution is a group of workflows that are dedicated to a specific business goal. A workflow can be added to multiple solutions. After you develop a solution and add a workflow to the solution, other users can reference and modify the workflow in their solutions for collaborative development.
Workflow
An abstract business entity and a collection of tasks, tables, resources, and functions for a specific business requirement. Tasks in this type of workflow are triggered to run as scheduled.
Manually triggered workflow
A collection of tasks, tables, resources, and functions for a specific business requirement.
Tasks in this type of workflow are manually triggered to run.
DAG
The abbreviation of
directed acyclic graph. A DAG is used to display nodes and their dependencies. In DataStudio, all tasks in a workflow are displayed in the same DAG. This facilitates task development and dependency configuration.Task
A basic execution unit of DataWorks. DataWorks runs tasks in sequence based on the dependencies between the tasks.
Node
A task in a DAG. DataWorks runs nodes in sequence based on the dependencies between the nodes.
Terms related to task scheduling
Term
Description
Dependency
Used to define the sequence in which tasks are run. If Node B can run only after Node A finishes running, Node A is the ancestor node of Node B, and Node B depends on Node A. In a DAG, dependencies are represented by arrows between nodes.
Output name
The identifier used to distinguish the current node from other nodes. An output name is globally unique. A node can contain multiple output names. Scheduling dependencies between nodes are configured based on output names.
Output table name
We recommend that you use the name of the table generated by the current task as the output table name. Proper configuration of an output table name can help check whether data is from an expected ancestor table when you configure dependencies for a descendant node. We recommend that you do not manually modify an output table name that is generated based on automatic parsing. The output table name serves only as an identifier. Modifying an output table name does not affect the name of the table that is actually generated by executing SQL statements. The name of an actually generated table is subject to the SQL logic.
NoteAn output name must be globally unique. However, no such limit is imposed on an output table name.
Resource group for scheduling
A resource group that is used for task scheduling. For more information about resource groups, see Overview.
Scheduling parameter
Configured for a node when the node is scheduled to run. The values of scheduling parameters are dynamically replaced at the scheduling time of the node. If you want to obtain information about the runtime environment, such as the date and time, during repeated running of code, you can dynamically assign values to variables in the code based on the definition of scheduling parameters in DataWorks.
Data timestamp
The previous day of the scheduling time (the time when you want to schedule the node). In offline computing scenarios, a data timestamp represents the date on which a business transaction is conducted. The value of a data timestamp is accurate to the day. For example, if you collect statistical data on the turnover of the previous day on the current day, the previous day is the date on which the business transaction is conducted and represents the data timestamp.
Scheduling time
The time when you want to schedule the task to process business data. The scheduling time is accurate to the second. The scheduling time can be different from the actual time at which the task is scheduled to run. The actual time at which a task is run is affected by multiple factors.