Auto triggered nodes are nodes that are periodically scheduled based on their scheduling configurations after they are committed to the scheduling system. You can view auto triggered nodes in a workspace and perform O&M operations on the desired auto triggered node in the list of auto triggered nodes on the Cycle Task page in Operation Center. For example, you can manually run or enable the scheduling system to automatically run the auto triggered node, view the details of the auto triggered node, or suspend or undeploy the auto triggered node. This topic describes the O&M operations that you can perform on an auto triggered node.
Usage notes
- Auto triggered nodes can be automatically scheduled to generate instances only in Operation Center in the production environment. Auto triggered nodes cannot be automatically scheduled to generate instances in Operation Center in the development environment.
- After you perform operations such as addition, modification, and undeployment on node code, scheduling configurations, resources, or functions in the production environment, you must commit and deploy the objects to make the configurations take effect.
- After you modify an auto triggered node, you must deploy the node. After the node is deployed, the modification takes effect in the production environment. The following issues may occur when you deploy the node: The node fails to be deployed, the deployment process is blocked, or the version of the deployed node does not meet your requirements. In these cases, we recommend that you go to the Cycle Task page in Operation Center to troubleshoot issues. For more information, see View and manage auto triggered nodes.
Overview
O&M operation | Description |
View auto triggered nodes | You can view the details of an auto triggered node and perform operations on the node in the list of auto triggered nodes or in the directed acyclic graph (DAG) of the node on the Cycle Task page. |
Run auto triggered nodes |
|
Manage auto triggered nodes |
|
View operation records of auto triggered nodes | You can view the operation logs, versions, and operation history of an auto triggered node. |
View auto triggered nodes
- Only the auto triggered nodes that are deployed to the production environment are displayed in the list of auto triggered nodes on the Cycle Task page in Operation Center.
- The following types of nodes are not automatically scheduled: nodes that do not depend on other nodes and nodes that are used as the ancestor nodes of other nodes and depend on their descendant nodes.
Run auto triggered nodes
You can understand the modes in which nodes in DataWorks are run and perform O&M diagnostics based on the running situations of nodes in an efficient manner.
Running modes
Instance type | Scenario | Relationship with an auto triggered node (How an instance is generated) | Instance trigger method (How an instance is triggered to run) |
Auto triggered node instance | Periodic extract, transform, and load (ETL) operations are required. | Every night, DataWorks generates auto triggered node instances that are scheduled to run on the next day based on the snapshot information of an auto triggered node at a specific point in time. Note Auto triggered nodes cannot be automatically scheduled to generate instances in Operation Center in the development environment. | DataWorks triggers an auto triggered node instance for running. |
Data backfill instance | You want to backfill data of a historical period of time or a period of time in the future for the current auto triggered node and its descendant nodes. This indicates that you must perform ETL operations on the data of that period of time. | You must backfill data for the current auto triggered node to generate data backfill instances for the node. | After you backfill data, the data backfill instances are generated and triggered to run. |
Test an auto triggered task and view test instances generated for the task | You want to test the current auto triggered node to check whether the node can be run as expected. Note The code of the auto triggered node is run during the test. | You must test the current auto triggered node to generate test instances for the node. | After you perform the test, the test instances are generated and triggered to run. |
Conditions for running an auto triggered node and issue troubleshooting
Conditions for running an auto triggered node
Troubleshooting for issues related to node running
Problem description and causes: The scheduling time of an auto triggered node arrives, but the node is not run. The possible causes of this issue may be that the ancestor nodes of the auto triggered node are still running until the scheduling time of the auto triggered node arrives, no sufficient resources can be used to run the auto triggered node, and the auto triggered node is frozen.
Emergency O&M operations
- If an auto triggered node depends on multiple ancestor nodes and one of the ancestor nodes is not run, you can find the instance generated for the auto triggered node on the Cycle Instance page, open the DAG of the instance, right-click the instance in the DAG, and then choose Note You must check whether this operation affects data output based on the code of the node for which the instance is generated and the lineage of the instance. . In the Delete Dependencies with Parent Nodes dialog box, you can specify the name of the ancestor node that is not run to remove the dependency between the auto triggered node and the ancestor node.
- If data quality issues occur on multiple descendant nodes of an auto triggered node, you can find the instance generated for the auto triggered node on the Cycle Instance page, open the DAG of the instance, right-click the instance in the DAG, and then choose Appendix: Forcefully rerun the descendant instances of an auto triggered node instance. or choose Emergency Operations > Force Rerun Descendant Nodes to forcefully rerun the auto triggered node and its descendant nodes. For more information, see
Manage auto triggered nodes
Deploy auto triggered nodes
Undeploy auto triggered nodes
If you no longer require an auto triggered node or a workflow, you can undeploy the node or all nodes in the workflow. After you undeploy an auto triggered node, you cannot find the node on the Cycle Task page. For more information, see Undeploy nodes.
Freeze auto triggered nodes
Operation | Scenario | Description |
Freeze an auto triggered node | If an auto triggered node and its descendant nodes do not need to be run for a specific period of time, you can freeze the auto triggered node. |
|
Freeze an instance | If an instance generated for an auto triggered node does not need to be run, you can freeze the instance. | The freeze operation takes effect only on the current instance. Other instances that are generated on the same day as the current instance and the instances that are generated later than the current day are not affected. |
Set Recurrence to Dry Run for an auto triggered node | If an auto triggered node does not need to be run for a specific period of time, but you do not want the running of the descendant nodes of the node to be blocked, you can set the Recurrence parameter to Dry Run for the auto triggered node. | The auto triggered node in the dry-run state generates dry-run instances. The system does not run the dry-run instances to generate data, does not generate run logs for the dry-run instances, and does not display running duration for the dry-run instances. Note The operation of setting the Recurrence parameter to Skip Execution for an auto triggered node on the DataStudio page achieves the same effect as the operation of freezing the auto triggered node in Operation Center. The modifications to an auto triggered node on the DataStudio page take effect only in the development environment. If you want the modifications to take effect in the production environment, you must deploy the auto triggered node to the production environment. For more information, see Deploy nodes. |
Manage node priorities
You can use the baseline management feature to adjust the priority of an auto triggered node in a baseline. Scheduling resources are preferentially allocated to the auto triggered nodes with higher priorities. For more information, see Manage baselines.
Configure monitoring and alerting for an auto triggered node
- You can configure a custom alert rule to monitor the status of an auto triggered node. For more information, see Manage custom alert rules.
- You can configure a data quality monitoring rule for an auto triggered node to monitor the table data that is generated when auto triggered node instances, data backfill instances, or test instances generated for the node are run. For more information, see Overview.
- You can configure a custom alert rule to monitor a resource group used to run an auto triggered node based on the number of instances that are generated for the node and waiting for resources in the resource group or based on the resource usage of the resource group. For more information, see Manage custom alert rules.
Change the resource group used to run an auto triggered node
Change the node owner
After Change Node Owner by RAM User is turned on, the workspace administrator can perform the following operations:
- Change the owner of a node or the owners of multiple nodes at a time on the DataStudio page.
- Change the owner of a node: The workspace administrator can change the owner of a node in the General section of the Properties tab on the DataStudio page.
- Change the owners of multiple nodes at a time: The workspace administrator can change the owners of multiple nodes at a time on the DataStudio page. For more information, see Perform operations on multiple DataWorks objects at a time.
After you change the owner of a node in the development environment, you must deploy the node to the production environment. This way, the change can take effect.
- Change the owner of a node or the owners of multiple nodes at a time in the production environment.
Operation Illustration Change the owner of a node Change the owners of multiple nodes at a time Note After you change the owner of a node in the production environment, the owner of the node in the development environment is also changed.
View operation records of auto triggered nodes
Entry point | Description | Illustration |
Operation Log tab of the DAG page of an auto triggered node or auto triggered node instance | You can view the operation records of an auto triggered node or auto triggered node instance on the Operation Log tab. | |
Versions tab on the DataStudio page | If no details about an operation that is performed on and deployed for an auto triggered node are recorded, you can go to the configuration tab of the auto triggered node and compare an existing version of the node with the version of the node in the production environment to obtain details about version changes. For more information, see Deploy nodes. | |
Operation History page in Operation Center | You can go to the Operation History page in Operation Center to view the operation records of an auto triggered node, an auto triggered node instance, or a baseline. For more information, see View operation records in Operation Center. |
FAQ
For more information about the FAQ for the O&M operations that can be performed on an auto triggered node, see FAQ about auto triggered nodes.