DataWorks:Create and use a data quality monitoring node

In the Monitoring Rules section of the configuration tab of the data quality monitoring node, click Select Table. In the Select Table panel, specify filter conditions to search for and select the table that you want to monitor.

• Non-partitioned table: By default, all data in a table is monitored. You can skip this configuration.

• Partitioned table: You must select the partition whose data you want to monitor. You can use scheduling parameters to specify the partition. You can click Preview to check whether the calculation result of the partition filter expression that you specified meets your expectations.

You can create a monitoring rule or import an existing monitoring rule. By default, the configured rules are enabled.

• Create a monitoring rule

  Click Create Rule to create a monitoring rule based on a template or a custom SQL statement.

  Method 1: Create a monitoring rule based on a built-in rule template

  DataWorks provides various built-in rule templates that you can use to create a data quality monitoring rule. The following figure shows the procedure.

  Note

  You can also find the desired template in the built-in rule template list on the left side of the Create Rule panel and click + Use to create a monitoring rule.

  

  Parameters for configuring a rule based on a built-in rule template

  Parameter	Description
  Rule Name	The name of the monitoring rule.
  Template	The template used to create the monitoring rule. Data Quality provides a large number of built-in table-level and field-level rule templates that are ready for use. For more information, see Built-in monitoring rule templates. Note You can configure field-level monitoring rules of the following types only for numeric fields: average value, sum of values, minimum value, and maximum value.
  Rule Scope	The application scope of the rule. For a table-level monitoring rule, the application scope is the current table by default. For a field-level monitoring rule, the application scope is a specific field.
  Comparison Method	The comparison method that is used by the rule to check whether the table data is as expected. Manual Settings: You can configure the comparison method to compare the data output result with the expected result based on your business requirements. You can select different comparison methods for different rule templates. You can view the comparison methods that are supported by a rule template in the DataWorks console. Numeric results: You can compare a numeric result with a fixed value, which is the expected value. The following comparison methods are supported: greater than, greater than or equal to, equal to, not equal to, less than, and less than or equal to. You can configure the normal data range (normal threshold) and abnormal data range (critical threshold) based on your business requirements. Fluctuation results: You can compare a fluctuation result with a fluctuation range. The following comparison methods are supported: absolute value, rise, and drop. You can configure the normal data range (normal threshold) based on your business requirements. You can also define data output exceptions (warning threshold) and unexpected data outputs (critical threshold) based on the degree of abnormal deviation. Intelligent Dynamic Threshold: If you select this option, you do not need to manually configure the fluctuation threshold or expected value. The system automatically determines the reasonable threshold based on intelligent algorithms. If abnormal data is detected, an alert is immediately triggered or the related task is immediately blocked. When the Comparison Method parameter is set to Intelligent Dynamic Threshold, you can configure the Degree of importance parameter. Note Only monitoring rules that you configure based on a custom SQL statement, a custom range, or a dynamic threshold support the intelligent dynamic threshold comparison method.
  Monitoring Threshold	If you set the Comparison Method parameter to Manual Settings, you can configure the Normal threshold and Red Threshold parameters. Normal threshold: If the data quality check result meets the specified condition, the data output is as expected. Red Threshold: If the data quality check result meets the specified condition, the data output is not as expected. If the rule that you configure is a rule of the fluctuation type, you must configure the warning threshold. Warning Threshold: If the data quality check result meets the specified condition, the data is abnormal but your business is not affected.
  Retain problem data	If the monitoring rule is enabled and a data quality check based on the rule fails, the system automatically creates a table to store the problematic data that is identified during the data quality check. Important The Retain problem data parameter is available only for MaxCompute tables. The Retain problem data parameter is available only for specific monitoring rules in Data Quality. If you turn off Status for the monitoring rule, problematic data is not stored.
  Status	Specifies whether to enable the rule in the production environment. Important If you turn off the switch for the rule, the rule cannot be triggered to perform a test run or triggered by the associated scheduling nodes.
  Degree of importance	The strength of the rule in your business. Strong rules are important rules. If you set the parameter to Strong rules and the critical threshold is exceeded, the scheduling node that you associate with the monitor is blocked by default. Weak rules are regular rules. If you set the parameter to Weak rules and the critical threshold is exceeded, the scheduling node that you associate with the monitor is not blocked by default.
  Description	You can add additional descriptions to the rule.

  Method 2: Create a monitoring rule based on a custom rule template

  Before you use this method, you must perform the following steps to create a custom rule template: Go to the Data Quality page. In the left-side navigation pane, choose Quality Assets > Rule Template Library. In the Templates section of the page that appears, click the plus icon to create a custom rule template. Then, you can create a monitoring rule based on the rule template. For more information, see Create and manage custom rule templates.

  The following figure shows how to create a monitoring rule based on a custom rule template.

  Note

  You can also find the desired template in the custom rule template list on the left side of the Create Rule panel and click + Use to create a monitoring rule.

  

  Parameters for configuring a rule based on a custom rule template

  Only the parameters that are unique to rules based on custom rule templates are described in the following table. For information about other parameters, see the parameters for configuring a rule based on a built-in rule template.

  Parameter	Description
  FLAG parameter	The SET statement that you want to execute before the SQL statement in the rule is executed.
  SQL	The SQL statement that determines the complete check logic. The returned results must be numeric and consist of one row and one column. In the custom SQL statement, enclose the partition filter expression in brackets []. Example: select count(*) from ${tableName} where ds=$[yyyymmdd]; Note In this statement, the value of the ${tableName} variable is dynamically replaced with the name of the table for which you are configuring monitoring rules. For information about how to configure a partition filter expression, see the Appendix 2: Built-in partition filter expressions section in this topic. If you have created a monitor for the table, the setting of the table partition that you specify in the Data Range parameter during the monitor configuration no longer takes effect for the table after you configure this parameter. The rule determines the table partition to be checked based on the setting of WHERE in the SQL statement.

  Method 3: Create a monitoring rule based on a custom SQL statement

  This method allows you to configure custom data quality check logic for tables.

  

  Parameters for configuring a rule based on a custom SQL statement

  Only the parameters that are unique to rules based on a custom SQL statement are described in the following table. For information about other parameters, see the parameters for configuring a rule based on a built-in rule template.

  Parameter	Description
  FLAG parameter	The SET statement that you want to execute before the SQL statement in the rule is executed.
  SQL	The SQL statement that determines the complete check logic. The returned results must be numeric and consist of one row and one column. In the custom SQL statement, enclose the partition filter expression in brackets []. Example: select count(*) from <table_name> where ds=$[yyyymmdd]; Note You must replace <table_name> with the name of the table for which you are configuring monitoring rules. The SQL statement determines the table that needs to be monitored. For information about how to configure a partition filter expression, see the Appendix 2: Built-in partition filter expressions section in this topic. If you have created a monitor for the table, the setting of the table partition that you specify in the Data Range parameter during the monitor configuration no longer takes effect for the table after you configure this parameter. The rule determines the table partition to be checked based on the setting of WHERE in the SQL statement.

• Import an existing monitoring rule

  If you already created monitoring rules for the selected table in Data Quality, you can import the rules to clone the rules. If you did not create monitoring rules for the table, you can create monitoring rules for the table in Data Quality. For more information, see Configure a monitoring rule for a single table.

  Note

  You can import multiple rules at a time and configure monitoring rules for fields in a table.

  Click Import Rule. In the Batch Import panel, you can specify filter conditions, such as the rule ID or name, rule template, and association range, to search for and select the rules that you want to import. The association range specifies the range of data that you want to monitor, which can be the entire table or specific fields in the table.

  

Select the resources required to run the rules. This means that you must select the data source in which you want to run the related data quality monitoring task. By default, the data source to which the monitored table belongs is selected.

The following table describes the categories of identified exceptions.

You can configure a policy to handle the exceptions that are identified based on the monitoring rules.

• Do not ignore: Stop the current node and set the node status to Failed when a specific exception is identified on the node. For example, you can use this policy to handle the exception that the critical threshold of a strong monitoring rule is exceeded.

  Note

  • If the current node fails to run, the nodes that depend on the current node do not run. This blocks the production link and prevents the spread of dirty data.

  • You can add multiple exception categories for detection.

  • You can use this policy when an exception has a large impact and blocks the running of descendant nodes.

• Ignore: Ignore the exception and continue to run the descendant nodes.

You can specify a method to receive information about exceptions, such as by email. When an exception is identified, DataWorks pushes information about the exception by using the specified method. This way, the related personnel can handle the exception at the earliest opportunity.