Configure a monitoring rule for multiple tables based on a template - DataWorks

Data Quality provides a variety of built-in table-level and field-level rule templates. This topic describes how to configure a monitoring rule based on a template.

Limits

You can configure monitoring rules based on templates for MaxCompute, E-MapReduce (EMR), Hologres, and AnalyticDB for PostgreSQL data sources.

Procedure

You can perform the following steps to configure a monitoring rule based on a template:

Select a rule template and configure the rule parameters.
Built-in rule templates are classified into table-level and field-level rule templates. After you select a template, you can configure a rule based on the template to check the data quality of a table. You can use the monitoring rule to determine whether the table data is as expected.
Add multiple tables or fields whose data quality needs to be checked by the rule at the same time.
You can select tables or fields whose data quality needs to be checked and apply the rule to the tables or fields.
Associate the rule with a new monitor or an existing monitor.
You can associate the rule with a new monitor or an existing monitor to specify the range of data whose quality you want to monitor based on the rule. If you want to monitor the data quality of a partitioned table, the data range is a partition.

Procedure

Step 1: Go to the Batch Add Monitoring Rules page

Log on to the DataWorks console. In the left-side navigation pane, choose Data Modeling and Development > Data Quality. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Quality.
In the left-side navigation pane, choose Rule configuration > Configure by Template.
Data Quality provides built-in table-level rule templates and field-level rule templates. You can find the desired template and click Configure Monitoring Rules to the right of the template to configure a monitoring rule based on the template for multiple tables or fields at the same time.

Step 2: Configure the attributes of the monitoring rule

On the Configure by Template page, find the template that you want to use and click Configure Monitoring Rules in the Actions column to go to the Batch Add Monitoring Rules wizard.

Configure basic attributes for the monitoring rule.

Parameter	Description
Data Source Type	The type of the data source of tables for which you want to configure a monitoring rule. Note You can configure monitoring rules based on templates for MaxCompute, E-MapReduce (EMR), Hologres, and AnalyticDB for PostgreSQL data sources.
Rule Source	The value of this parameter is fixed as Built-in Template.
Template	The name of the built-in rule template. For more information, see Built-in monitoring rule templates.
Rule Name	The naming format of the monitoring rule. The name of the monitoring rule is automatically generated. You can change the suffix in the naming format based on your business requirements.
Description	The description of the monitoring rule.

Configure advanced attributes for the monitoring rule.
Click Next to go to the Generate Monitoring Rule step.

Step 3: Add multiple tables or fields whose data quality needs to be checked by the rule at the same time

You can add multiple tables or fields whose data quality needs to be checked based on the selected table-level or field-level rule template.

Add tables

Click Add Table.
In the Batch Create panel, select the tables for which you want to configure a monitoring rule. All MaxCompute tables are displayed in the panel. You can also enter a keyword to search for the desired tables.
After you select tables, click Confirm to add the tables to the monitored table list.

Add fields

Click Add Fields.
In the Select Field dialog box, select the tables that contain the fields for which you want to configure a monitoring rule. All MaxCompute tables are displayed in the Table to be selected section.
After you select tables, the Select Fields section displays all fields in the selected tables. You can filter the fields by field name or field description.
Select the fields for which you want to configure a monitoring rule and click Create to add the fields to the monitored field list.

Step 4: Associate the rule with a new monitor or an existing monitor

You can associate the rule with a new monitor or an existing monitor to specify the range of data whose quality you want to monitor based on the rule. If you want to monitor the data quality of a partitioned table, the data range is a partition.

You can separately configure a monitor for each table or field or configure a monitor for multiple tables or fields at the same time.

Configure a monitor for multiple tables or fields at the same time

Select multiple tables or fields and click Set Quality Inspection Task.
Select Automatic Association, Unassociate, or Batch fast add from the Set Quality Inspection Task drop-down list.
- Automatic Association: The selected tables or fields are automatically associated with an existing monitor.
- Unassociate: The selected tables or fields are disassociated from a monitor.
- Batch fast add: You can configure the data range and running settings for the selected tables or fields.

Separately configure a monitor for each table or field

Associate the monitoring rule with a monitor. To associate the rule with a monitor, find the desired table or field, click the drop-down list in the Quality monitoring column, and then select an existing monitor.
If no monitor is available, select New Quality Inspection Task from the drop-down list to create a monitor and associate the rule with the created monitor. The following table describes the parameters.

Step 5: Test the rule

Click Generate Monitoring Rule to go to the Verify Monitoring Rule step. On the Verify Monitoring Rule page, you can perform the following operations:
- Verify the reasonableness of the rule configurations: Perform a test run.
  After the rules of the selected tables are created, you can select one or more rules to perform a test run. In the Test Run dialog box, configure the Scheduling time parameter, which specifies the simulated triggering time of the rule. The system calculates the value of a partition key in a partitioned table that you want to monitor based on the specified simulated triggering time and the value of the Data Range parameter configured when you associate the rule with a monitor. Click Test Run to check whether the data in the partition of the table complies with the configured monitoring rule.
  After the test run is complete, you can click Running records in the Actions column of the rule to view the details of the test run and perform related operations.
- Specify the alert recipient: Subscribe to a rule.
  You can find the rule of a table and click Subscriptions in the Actions column to subscribe to the rule. In the Alert Subscription dialog box, you can configure the Notification Method and Recipient parameters. If an alert is triggered, the specified recipient is notified by using one of the following notification methods: Email, Email and SMS, DingTalk Chatbot, DingTalk Chatbot @ALL, Lark Group Chatbot, Enterprise WeChat Chatbot, Custom Webhook, and Telephone.
  Note
  The Custom Webhook notification method is supported only in DataWorks Enterprise Edition. For information about the message format of an alert notification sent by using a custom webhook URL, see the "Appendix: Message format of alert notifications sent by using a custom webhook URL" section in Configure a monitoring rule for multiple tables based on a template.
- Specify a trigger method for a rule: Associate a rule with scheduling nodes.
  You can click Recommend Set Run Mode or Manually set the operation mode to associate one or more monitoring rules with the auto triggered nodes that generate table data. In Operation Center, auto triggered nodes generate the table data after the auto triggered node instances, data backfill instances, or test instances generated for the auto triggered nodes are successfully run. When the auto triggered nodes start to run, the monitoring rules are triggered. You can specify the strength of a rule to control whether to block the running of the descendant nodes of an associated auto triggered node. This helps reduce the impact of dirty data records.
  - Recommend Set Run Mode: The system associates the selected monitoring rules with auto triggered nodes based on the lineages of the auto triggered nodes that generate the table data.
  - Manually set the operation mode: You can manually associate the selected monitoring rule with specific auto triggered nodes.
  Important
  A monitoring rule can be triggered only when the rule is associated with related auto triggered nodes.
- Delete a rule: You can select one or more rules that you want to delete.
- View the details of a rule: You can click View Rule Details in the Actions column of a rule to view the rule details. On the page that appears, you can perform the following operations: modify, start, stop, and delete the rule, specify the rule strength, and view logs.
After the test on the monitoring rule is complete and the monitoring rule is associated with auto triggered nodes, click Save Run Mode.

What to do next

After the monitor is run, you can choose Maintenance > Monitoring in the left-side navigation pane of the Data Quality page to view the quality check status of the specified table and choose Maintenance > Running Records to view the complete check records of the rule.

Appendix: Message format of alert notifications sent by using a custom webhook URL

The following section describes the message format of an alert notification sent by using a custom webhook URL and the related parameters.

Sample message

{
  "detailUrl": "https://dqc-cn-zhangjiakou.data.aliyun.com/?defaultProjectId=3058#/jobDetail?envType=ODPS&projectName=yongxunQA_zhangbei_standard&tableName=sx_up_001&entityId=10878&taskId=16876941111958fa4ce0e0b5746379cd9bc67999d05f8&bizDate=1687536000000&executeTime=1687694111000",
  "datasourceName": "emr_test_01",
  "engineTypeName": "EMR",
  "projectName": "Project name",
  "dqcEntityQuality": {
    "entityName": "tb_auto_test",
    "actualExpression": "ds=20230625",
    "strongRuleAlarmNum": 1,
    "weakRuleAlarmNum": 0
  },
  "ruleChecks": [
    {
      "blockType": 0,
      "warningThreshold": 0.1,
      "property": "id",
      "tableName": "tb_auto_test",
      "comment": "Test a monitoring rule",
      "checkResultStatus": 2,
      "templateName": "Compare the Number of Unique Field Values Against Expectation",
      "checkerName": "fulx",
      "ruleId": 123421,
      "fixedCheck": false,
      "op": "",
      "upperValue": 22200,
      "actualExpression": "ds=20230625",
      "externalId": "123112232",
      "timeCost": "10",
      "trend": "up",
      "externalType": "CWF2",
      "bizDate": 1600704000000,
      "checkResult": 2,
      "matchExpression": "ds=$[yyyymmdd]",
      "checkerType": 0,
      "projectName": "auto_test",
      "beginTime": 1600704000000,
      "dateType": "YMD",
      "criticalThreshold": "0.6",
      "isPrediction": false,
      "ruleName": "Rule name",
      "checkerId": 7,
      "discreteCheck": true,
      "endTime": 1600704000000,
      "MethodName": "max",
      "lowerValue": 2344,
      "entityId": 12142421,
      "whereCondition": "type!='type2'",
      "expectValue": 90,
      "templateId": 5,
      "taskId": "16008552981681a0d6",
      "id": 234241453,
      "open": true,
      "referenceValue": [
        {
          "discreteProperty": "type1",
          "value": 20,
          "bizDate": "1600704000000",
          "singleCheckResult": 2,
          "threshold": 0.2
        }
      ],
      "sampleValue": [
        {
          "discreteProperty": "type2",
          "bizDate": "1600704000000",
          "value": 23
        }
      ]
    }
  ]
}

The following table describes the parameters in the preceding code.

Parameter	Type	Sample value	Description
ProjectName	String	autotest	The name of the compute engine instance or data source whose data quality is monitored.
actualExpression	String	ds=20200925	The partition in the monitored data source table.
RuleChecks	Array of RuleChecks		The returned monitoring results.
BlockType	Integer	1	The strength of the monitoring rule. The strength of a monitoring rule specifies the importance of the rule. Valid values: 1: the monitoring rule is a strong rule. 0: the monitoring rule is a weak rule. You can specify the strength of a monitoring rule based on your business requirements. If a monitoring rule is a strong rule and the critical threshold is exceeded, a critical alert is reported and tasks that are associated with the rule are blocked from running.
WarningThreshold	Float	0.1	The threshold for a warning alert. The threshold specifies the deviation of the monitoring result from the expected value. You can specify a custom value for the threshold based on your business requirements.
Property	String	type	The field whose data quality is checked based on the monitoring rule. This field is a column in the data source table that is monitored.
TableName	String	dual	The name of the table that is monitored.
Comment	String	Description of the monitoring rule	The description of the monitoring rule.
CheckResultStatus	Integer	2	The status of the monitoring result.
TemplateName	String	Compare Number of Unique Field Values Against Expectation	The name of the monitoring template.
CheckerName	String	fulx	The name of the checker.
RuleId	Long	123421	The ID of the monitoring rule.
FixedCheck	Boolean	false	Indicates whether the monitoring is performed based on a fixed value. Valid values: true false
Op	String	>	The comparison operator.
UpperValue	Float	22200	The upper limit of the predicted result. The value of this parameter is automatically generated based on the threshold that you specify.
ActualExpression	String	ds=20200925	The partition in the monitored data source table.
ExternalId	String	123112232	The ID of the node.
TimeCost	String	10	The time consumed to run the monitoring task.
Trend	String	up	The trend of the monitoring result.
ExternalType	String	CWF2	The type of the scheduling system. Only CWF scheduling systems are supported.
BizDate	Long	1600704000000	The data timestamp. In most cases, if the monitored business entity is offline data, the value is one day before the monitoring is performed.
CheckResult	Integer	2	The monitoring result.
MatchExpression	String	ds=$[yyyymmdd]	The partition filter expression.
CheckerType	Integer	0	The type of the checker.
ProjectName	String	autotest	The name of the compute engine instance or data source whose data quality is monitored.
BeginTime	Long	1600704000000	The time when the monitoring started.
DateType	String	YMD	The scheduling frequency. In most cases, the value of this parameter is YMD. The value indicates year, month, and day.
CriticalThreshold	Float	0.6	The threshold for a critical alert. The threshold specifies the deviation of the monitoring result from the expected value. You can specify a custom value for the threshold based on your business requirements. If a monitoring rule is a strong rule and the critical threshold is exceeded, a critical alert is reported and tasks that are associated with the rule are blocked from running.
IsPrediction	Boolean	false	Specifies whether the monitoring result is the same as the predicted result. Valid values: true false
RuleName	String	Test	The name of the monitoring rule.
CheckerId	Integer	7	The ID of the checker.
DiscreteCheck	Boolean	true	Specifies whether the monitoring is discrete monitoring. Valid values: true false
EndTime	Long	1600704000000	The time when the monitoring ended.
MethodName	String	max	The method used to collect sample data, such as avg, count, sum, min, max, count_distinct, user_defined, table_count, table_size, table_dt_load_count, table_dt_refuseload_count, null_value, null_value/table_count, (table_count-count_distinct)/table_count, or table_count-count_distinct.
LowerValue	Float	2344	The lower limit of the predicted result. The value of this parameter is automatically generated based on the threshold that you specify.
EntityId	Long	14534343	The ID of the partition filter expression.
WhereCondition	String	type!='type2'	The filter condition of the monitoring task.
ExpectValue	Float	90	The expected value.
TemplateId	Integer	5	The ID of the monitoring template.
TaskId	String	16008552981681a0d6****	The ID of the monitoring task.
Id	Long	2231123	The ID of the primary key.
ReferenceValue	Array of ReferenceValue		The historical sample values.
DiscreteProperty	String	type1	The values of the sample field that are grouped by using the GROUP BY clause. For example, the values of the Gender field are grouped by using the GROUP BY clause. In this case, the values of the DiscreteProperty parameter are Male, Female, and null.
Value	Float	20	The sample value.
BizDate	String	1600704000000	The data timestamp. In most cases, if the monitored business entity is offline data, the value is one day before the monitoring is performed.
SingleCheckResult	Integer	2	The string of the monitoring result.
Threshold	Float	0.2	The threshold.
SampleValue	Array of SampleValue		The current sample values.
DiscreteProperty	String	type2	The values of the sample field that are grouped by using the GROUP BY clause. For example, the values of the Gender field are grouped by using the GROUP BY clause. In this case, the values of the DiscreteProperty parameter are Male, Female, and null.
BizDate	String	1600704000000	The data timestamp. In most cases, if the monitored business entity is offline data, the value is one day before the monitoring is performed.
Value	Float	23	The sample value.
Open	Boolean	true	Specifies whether the monitoring rule is enabled.