A dependency on the previous cycle means that a node's current instance depends on the instance of a specific node from the previous cycle. The current instance runs only after the instance from the previous cycle runs successfully. You can set a cross-cycle dependency if a node needs to use data produced by an upstream node on the previous day, or if an hourly or minutely task needs to depend on its own instance from the previous hour or minute. This topic describes how to configure cross-cycle dependencies and the types of dependencies.
Precautions
Note the following items when you configure dependencies on the previous cycle.
Category | Description | Related information |
Display of cross-cycle dependencies | Cross-cycle dependencies are shown as dashed lines in the Directed Acyclic Graph (DAG) in DataWorks. | |
Confirm whether same-cycle dependencies are needed after you set a cross-cycle dependency | After you configure dependencies, a downstream node runs only after all its ancestor nodes run successfully. The automatic parsing feature for same-cycle dependencies is enabled by default. If you set a cross-cycle dependency, confirm whether a same-cycle dependency is also needed. If not, delete the automatically parsed same-cycle dependency to ensure the downstream node runs as expected. | |
When to set cross-cycle dependencies in complex scenarios | In some scenarios, if same-cycle dependencies do not meet your requirements, you can set cross-cycle dependencies. For example, if a daily node depends on an hourly node, the daily node depends on all instances of the hourly node for that day by default. You can configure a self-dependency for the hourly node. This allows the daily node to depend on the hourly node's instance from a specific cycle. | Must-read: Principles and examples of scheduling configurations for complex dependency scenarios |
Preview node dependencies | To prevent scheduling delays in the production environment caused by unexpected dependencies, preview the dependencies before you publish the task. This confirms that the instance relationships for each cycle are correct. | |
Task publishing instructions | Ensure that both the upstream and downstream nodes with a cross-cycle dependency are published to the production environment. After publishing, you can view the dependency in the Operation Center for that environment. |
Entry point
Go to the node editing page in Data Studio. In the navigation pane on the right, click Scheduling Configuration. In the section, you can configure the node's dependencies.
Types of dependencies
The following table describes the supported types of dependencies on the previous cycle.
Dependency type | Node dependency | Business scenario |
Dependency on the previous cycle: Current node (self-dependency) | The current instance of a node depends on the result of its own instance from the previous cycle. The current instance runs only if the instance from the previous cycle ran successfully. | The current instance of a node depends on the business data produced by its own instance from the previous cycle. |
The current instance of a node depends on the status of its downstream nodes' instances from the previous cycle. The current instance runs only if the instances of its downstream nodes from the previous cycle ran successfully. | The current instance of a node depends on whether its downstream nodes successfully processed the data in its sink table (output table) from the previous cycle. | |
The current instance of a node depends on the result of another node's instance from the previous cycle. The current instance runs only if the instance of the other node from the previous cycle ran successfully. | The current instance of a node logically depends on data from another business process, but the node itself does not perform operations on that data. |
Dependency on the previous cycle: Current node (self-dependency)
The current instance of a node depends on the business data produced by its own instance from the previous cycle. The following figure shows an example of the dependency settings and the resulting relationship.
If a daily task depends on an hourly or minutely task, the running time of the daily task is affected by whether a self-dependency is set for the hourly or minutely task:
Hourly or minutely task without a self-dependency
By default, the daily task depends on all instances of the hourly or minutely task for that day. The daily task then aggregates and processes all table data produced by the hourly or minutely task for that day.
Hourly or minutely task with a self-dependency
The daily task attaches to the hourly or minutely instance with the closest scheduled time, based on the principle of scheduling time proximity. It does not depend on all hourly or minutely instances for that day.
For more information about specific dependencies, see Appendix: Summary of complex dependency scenarios.
When you backfill data for a task with a self-dependency, the data instances may run sequentially in chronological order. For more information, see Data backfill for tasks with a self-dependency.
Dependency on the previous cycle: First-level child nodes
The current instance of a node depends on whether its downstream nodes successfully processed its table data from the previous cycle.
For example, Node C has two downstream nodes: Node A and Node B. A dependency on first-level child nodes means that Node C depends on the results of Node A and Node B from the previous cycle. In the figure, the current cycle is T and the previous cycle is T-1. The current instance of Node C runs only after the instances of Node A and Node B from the previous cycle run successfully.
Dependency on the previous cycle: Other nodes
The current node's instance runs only if another node's instance from the previous cycle ran successfully.
For example, Node A and Node B are two downstream nodes of Node C. A dependency on other nodes means that Node B depends on the result of Node D from the previous cycle. In the figure, the current cycle is T and the previous cycle is T-1. The current instance of Node B runs only after the instance of Node D from the previous cycle runs successfully.
Inherit the dry-run property from upstream nodes
This configuration is typically used with branch nodes.
Entry point
In the Scheduling Dependency section, you can set Skip Dry-Run Property of Upstream Node to Yes. This setting prevents the downstream node from being affected by the dry-run property of this node from the previous cycle.
Application Scenario
A node has multiple downstream nodes. During task execution, some downstream nodes are set to the dry-run state. If a dry-run downstream node is configured with a self-dependency on its previous cycle, its dry-run property is inherited by its child nodes. This causes the node to remain in a dry-run state continuously. If you do not want child nodes to inherit the dry-run property of the upstream node, set Skip Dry-Run Property of Upstream Node to Yes in the Scheduling Dependency section.
Example scenario
Assume that Assign_Node is an assignment node, Branch_Node is a branch node, and Shell_Node1 and Shell_Node2 are downstream nodes of Branch_Node. All these nodes are scheduled to run daily.
During runtime, Shell_Node1 is set to dry-run, and Shell_Node2 runs normally.
Node Shell_Node1 is configured with a self-dependency on its previous cycle.
In the current cycle (T), the recurring instance generated for node Shell_Node1 is named
Shell_Node1'.In the previous cycle (T-1), the recurring instance generated for node Shell_Node1 was named
Shell_Node1.
The recurring instance of Shell_Node1in the current epoch (T) depends on the recurring instance ofShell_Node1from the previous epoch (T-1) to run. The `Shell_Node1` node always performs a dry-run because it is a descendant node and inherits the `dry-run` property from its ancestor node.
Preview dependencies
After you complete the configuration, you can preview the node's dependencies. For more information, see Next: Confirm that the dependencies meet your expectations.