DataWorks:Data development: Developers

Go to the DataStudio page

Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.

Procedure

1. Step 1: Create a workflow

   Code is developed based on a workflow in DataStudio. Before you perform development operations, you must create a workflow.

2. Step 2: Create tables

   DataWorks allows you to create tables in a visualized manner and displays tables in a directory structure. Before data development, you must create a table in the MaxCompute compute engine to store the data processing results.

3. Step 3: Create a node

   Data development in DataWorks is based on nodes, and nodes of different types of compute engines are encapsulated into different types of nodes in DataWorks. You can select a suitable node type to develop a compute engine node based on your business requirements.

4. Step 4: Configure the node

   You can write code for the node on the node configuration tab based on the syntax that is supported by the related database.

5. Step 5: Configure scheduling properties for the node

   You can configure scheduling properties for the node to enable the system to periodically schedule and run the node.

6. Step 6: Debug the code of the node

   You can use the quick run feature for code snippets, or the Run feature or Run with Parameters feature to debug and check the logic of the code of the node.

7. Step 7: Save and commit the node

   After the node is debugged, you must save and commit the node.

8. Step 8: Perform smoke testing

   To ensure efficient running of a node in the production environment and prevent wastage of computing resources, you can commit the node to the development environment and perform smoke testing in the development environment before you deploy the node. This helps ensure the correctness of the code of the node.

9. Step 9: Deploy the node

   DataWorks can automatically schedule only nodes that are deployed to the production environment. After the node passes the smoke testing, you must deploy the node to the production environment to enable DataWorks to periodically schedule the node.

Step 1: Create a workflow

DataWorks organizes data development processes by using workflows. DataWorks provides dashboards for different types of nodes in each workflow and allows you to use tools and optimize and manage nodes on the dashboards. This facilitates data development and management. You can place nodes of the same business type in one workflow based on your business requirements.

1. Go to the DataStudio page.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.

2. Create a workflow.

   You can use one of the following methods to create a workflow:

   • Method 1: Move the pointer over the icon and click Create Workflow.

   • Method 2: Right-click Business Flow in the Scheduled Workflow pane and select Create Workflow.

3. In the Create Workflow dialog box, configure the Workflow Name and Description parameters for the workflow, and click Create.

   In this example, the Workflow Name parameter is set to Create the first auto triggered node. You can configure the Workflow Name parameter based on your business requirements in actual data development scenarios.

   Note

   For more information about how to use workflows, see Create a workflow.

Step 2: Create tables

A data development node of DataWorks cleanses and processes source data. Before data development, you must create a table in the required compute engine to store the data cleansing results and define the table schema.

1. Create tables.

   1. Click Business Flow in the Scheduled Workflow pane. Find the workflow that is created in Step 1, click the workflow name, right-click MaxCompute, and then select Create Table.

   2. In the Create Table dialog box, configure the parameters such as Engine Instance and Name parameters.

   In this example, the following tables are created.

   Table name	Description
   bank_data	Used to store raw business data.
   result_table	Used to store the data cleansing results.

   Note

   • For information about table creation statements, see Table creation statements.

   • For information about how to create tables in different compute engines in a visualized manner, such as creating a MaxCompute table or an EMR table, see Create tables.

2. Generate table schemas.

   Go to the configuration tabs of the tables, switch to the DDL mode, and use DDL statements to generate schemas for the tables. After the table schemas are generated, configure the Display Name parameter in the General section, click Commit to Development Environment, and then click Commit to Production Environment in the top toolbar. After the tables are committed, you can view the tables in the MaxCompute data source in the related environment. For information about how to view the data sources that are added to workspaces in different environments, see Add a MaxCompute data source.

   Note

   • Operations such as table creation and table update can take effect in the related compute engines only after they are committed to the required environment.

   • You can also follow the on-screen instructions that are displayed in the DataWorks console to configure the table schemas in a visualized manner based on your business requirements. For more information about how to create a table in a visualized manner, see Create and manage MaxCompute tables.

   In this example, the following statement is used to generate the schema of the bank_data table:

   CREATE TABLE IF NOT EXISTS bank_data
   (
    age             BIGINT COMMENT 'Age',
    job             STRING COMMENT 'Job type',
    marital         STRING COMMENT 'Marital status',
    education       STRING COMMENT 'Education level',
    default         STRING COMMENT 'Credit card',
    housing         STRING COMMENT 'Mortgage',
    loan            STRING COMMENT 'Loan',
    contact         STRING COMMENT 'Contact information',
    month           STRING COMMENT 'Month',
    day_of_week     STRING COMMENT 'Day of the week',
    duration        STRING COMMENT 'Duration',
    campaign        BIGINT COMMENT 'Number of contacts during the campaign',
    pdays           DOUBLE COMMENT 'Interval from the last contact',
    previous        DOUBLE COMMENT 'Number of contacts with the customer',
    poutcome        STRING COMMENT 'Result of the previous marketing campaign',
    emp_var_rate    DOUBLE COMMENT 'Employment change rate',
    cons_price_idx  DOUBLE COMMENT 'Consumer price index',
    cons_conf_idx   DOUBLE COMMENT 'Consumer confidence index',
    euribor3m       DOUBLE COMMENT 'Euro deposit rate',
    nr_employed     DOUBLE COMMENT 'Number of employees',
    y               BIGINT COMMENT 'Time deposit available or not'
   );

   In this example, the following statement is used to generate the schema of the result_table table:

   CREATE TABLE IF NOT EXISTS result_table
   (
   education STRING COMMENT 'Education level',
   num BIGINT COMMENT 'Number of persons'
   )
   PARTITIONED BY
   (
   day STRING,
   hour STRING
   );

3. Upload data.

   Upload raw business data to a table in DataWorks. In this example, data is uploaded to the bank_data table. In this example, a file named banking.txt is uploaded from an on-premises machine to DataWorks. The following figure shows the procedure. For more information about how to upload data, see Upload a file from your on-premises machine to the bank_data table.

Step 3: Create a node

1. Create a node.

   You can use one of the following methods to create a node:

   • Method 1: Create a node in the Scheduled Workflow pane

     1. In the Scheduled Workflow pane of the DataStudio page, click Business Flow, find the workflow that you created, and then click the name of the workflow.

     2. Right-click the compute engine that you want to use, and select a suitable node type after you move the pointer over Create Node to create a node of the selected type.

   • Method 2: Create a node on the configuration tab of the workflow

     1. In the Scheduled Workflow pane of the DataStudio page, click Business Flow and find the workflow that you created.

     2. Double-click the name of the workflow to go to the configuration tab of the workflow.

     3. In the left-side section of the configuration tab, click the required node type or drag the required node type to the canvas on the right side.

2. In the Create Node dialog box, configure the parameters such as Engine Instance and Name.

   In this example, an ODPS SQL node named result_table is created. The name of the node is the same as the name of the result table that is created in Step 2.

   Note

   When you use DataWorks for data development, you need to use a compute node to cleanse the data and then store the cleansing results in a result table. We recommend that you use the name of the result table as the name of the node to quickly locate the table data that is generated by the node.

   

Step 4: Configure the node

Find the node that you created in Step 3, and double-click the name of the node to go to the node configuration tab. On the node configuration tab, write the code of the node based on the syntax that is supported by the related database.

INSERT OVERWRITE TABLE result_table partition (day='${day}', hour='${hour}')
SELECT education
, COUNT(marital) AS num
FROM bank_data
GROUP BY education;

Step 5: Configure scheduling properties for the node

Step 6: Debug the code of the node

Step 7: Save and commit the node

1. Click the icon in the top toolbar to save the node.

2. Click the icon in the top toolbar to commit the node to the development environment.

Step 8: Perform smoke testing

Step 9: Deploy the node

In a workspace in standard mode, the operations that are committed on the DataStudio page, including addition, update, and deletion of data development nodes, resources, and functions, are in the pending state on the Create Deploy Task page. You can click Deploy to go to the Create Deploy Task page, and deploy the related operations to the production environment. The operations take effect only after they are deployed to the production environment. For more information, see Deploy nodes.