All Products
Search

This Product

    DataWorks:Configuration example: MaxCompute

    Document Center

    DataWorks:Configuration example: MaxCompute

    Last Updated:Nov 14, 2024

    Monitoring rules are the core of Data Quality. You can configure monitoring rules to monitor data in MaxCompute, E-MapReduce (EMR), Hologres, AnalyticDB for PostgreSQL, and AnalyticDB for MySQL. This topic describes how to configure monitoring rules for MaxCompute.

    Procedure

    Step 1: Add a MaxCompute data source

    1. Go to the Data Sources page.

      1. Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Management Center. On the page that appears, select the desired workspace from the drop-down list and click Go to Management Center.

      2. In the left-side navigation pane of the SettingCenter page, choose Data Sources > Data Sources.

    2. In the upper-left corner of the Data Sources page, click Add Data Source. In the Add Data Source dialog box, click MaxCompute. For more information, see Add a MaxCompute data source.

    3. Click the 图标 icon in the upper-left corner and choose All Products > Data Development And Task Operation > DataStudio.

    4. In the left-side navigation pane of the DataStudio page, click Data Source. On the Data Source page, associate the MaxCompute data source with DataStudio.

      Note

      In this example, the test data added to the test table is from a MaxCompute public dataset. In actual business scenarios, you must configure a monitor and monitoring rules for your business data.

    Step 2: Go to the Table Quality Details page

    1. Click the 图标 icon in the upper-left corner and choose All Products > Data Governance > Data Quality.

    2. In the left-side navigation pane, choose Configure Rules > Configure by Table.

    3. In the Connection section, select a database. On the right side of the Configure by Table page, find the table for which you want to configure monitoring rules and click the name of the table or click Configure Monitoring Rule in the Actions column. The Table Quality Details page appears.

      Note

      You can search for a table by name. You can use the initial letters of a table name to perform a fuzzy match.

      image

    Step 3: Create a monitor

    1. On the Rule Management tab of the Table Quality Details page, click the image icon next to Monitor Perspective. The Create Monitor page appears.

    2. Configure the parameters for quality monitoring.

      Note

      You can create a monitor based on the quality check requirements of your table. For information about how to create a monitor, see Configure a monitoring rule for a single table.

      image

    3. Click Save.

    Step 4: Configure monitoring rules

    1. On the Rule Management tab of the Table Quality Details page, click Create Rule.

    2. Create a monitoring rule. In this example, a built-in template is used to create a monitoring rule.

      Note

      You can create monitoring rules for your table based on your business requirements. For more information, see Configure a monitoring rule for a single table.

      image

    3. Add the monitoring rule to the monitor created in Step 3.

      image

    4. Click Determine.

    Step 5: Test the monitoring rule

    1. On the Monitor tab of the Table Quality Details page, find the created monitor and click Test in the Actions column.

      image

    2. In the Test Run dialog box, check the values of the Data Range and Scheduling Time parameters and click Test Run.

      image

    3. If Started is displayed, you can click View Details to view the results of the test run.

      In this example, the number of rows in the MaxCompute test table is greater than 2,000. In this case, the quality check result hits the critical threshold.

      image

    4. After you make sure that the monitoring rule can be properly triggered, you can change the value of the Trigger Method parameter to Triggered by Node Scheduling in Production Environment for the monitor. After the scheduling node that you associated with the monitor finishes running in Operation Center, the monitoring rule is automatically triggered.

    Step 6: Subscribe to the monitor

    1. On the Monitor tab of the Table Quality Details page, find the created monitor and choose More > Subscribe to Alerts in the Actions column.

    2. In the Alert Subscription dialog box, add a notification method and a recipient and click Save in the Actions column.

      image

    References