Configure and use scheduling parameters - DataWorks - Alibaba Cloud Documentation Center

Scheduling parameters are automatically replaced with specific values based on the data timestamps of nodes and value formats of scheduling parameters. This enables dynamic parameter configuration for node scheduling. This topic describes how to configure and use scheduling parameters. This topic also describes how to perform smoke testing to check whether the scheduling parameters configured for an ODPS SQL node work as expected.

Precautions

Before you click the or 高级运行 icon in the toolbar to run a node, you must manually assign constants to the variables that you want to reference in the code of the node. In this case, you cannot check whether the configured scheduling parameters work as expected.

Process of configuring scheduling parameters

The following table describes how to configure scheduling parameters.

Area	Description	References
1	Define variables in the `${Variable}` format in the code of a node and configure the scheduling parameters that you want to assign to the variables as values in the Parameters section of the Properties tab.	Define variables
2	Assign the scheduling parameters as values to the variables in the code of the node in a visualized manner or by using expressions.	Assign scheduling parameters as values to variables in the code of a node
3	Use the scheduling parameter preview feature to test whether the scheduling parameters are replaced as expected when the node is scheduled to run.	Test whether scheduling parameters are replaced as expected
4	Use the smoke testing feature in the development environment to test whether the code of the node is executed and the scheduling parameters are replaced as expected.	Test whether the code of the node is executed and the scheduling parameters are replaced as expected
5	After the test is successful, deploy the node to the production environment. On the Auto Triggered Nodes page in Operation Center in the production environment, check whether the scheduling parameters used by the node meet your business requirements.	View the configurations of scheduling parameters in Operation Center in the production environment

For information about a complete sample for configuring scheduling parameters and verifying the execution of the scheduling parameters, see Example on how to configure scheduling parameters. For some frequently asked questions about scheduling parameter configuration, see Scheduling parameters.

Define variables

You can perform the steps in the following figure to define variables in the code of a node and configure the scheduling parameters that you want to assign to the variables as values in the Parameters section of the Properties tab based on your business requirements.

Note

Some nodes use the built-in variable ${bizdate}. The built-in variable is automatically assigned $bizdate as a value. You can replace bizdate with a custom variable name based on your business requirements.

Assign scheduling parameters as values to variables in the code of a node

DataWorks allows you to add scheduling parameters in a visualized manner or by using expressions.

Method to define scheduling parameters	Feature	Description	Screenshot
Visualization mode	Add Parameter	You can click Add Parameter to configure multiple scheduling parameters for a node. For more information about scheduling parameters, see Supported formats of scheduling parameters.
Visualization mode	Load Parameters in Code	After you click Load Parameters in Code, DataWorks identifies the variable names defined in the code of the current node and adds the identified variable names to the Parameters section. Note In most cases, custom variables are defined in the format of `${Custom variable name}` in the code. For PyODPS nodes or common Shell nodes, variable names are defined by using a different method from that for other types of nodes. For more information about how to define custom variables for different types of nodes, see Configure scheduling parameters for different types of nodes.
Expression mode	Define by expression	By default, the system provides an interface on which you can configure scheduling parameters for a node. You can also click Define by expression to configure scheduling parameters by using an expression. Note When you use an expression to configure multiple scheduling parameters, separate multiple assignment equations with spaces. If you add, delete, or modify scheduling parameters by using expressions, DataWorks checks the syntax of the expressions. If the check fails, the scheduling parameters fail to be configured. For example, if a space exists on either side of the equal sign (=), DataWorks reports an error.

Test whether scheduling parameters are replaced as expected

Note

The value of a custom time parameter in the $[...] format is related to the scheduling frequency of a node. This indicates that the value of a $[...] parameter is related to the scheduling time of a node. For information about how to configure the scheduling frequency, see Configure time properties.

After you assign values to scheduling parameters, you can use the scheduling parameter preview feature to test whether the scheduling parameters are replaced as expected.

DataWorks replaces the scheduling parameters based on the scheduling configurations of the node when the node is scheduled to run in a test environment. You can check whether the scheduling parameters are replaced with valid values. If the values do not meet your business requirements, you can adjust the scheduling configurations of the node at the earliest opportunity to prevent the invalid configurations from affecting the running of the node.

Test whether the code of the node is executed and the scheduling parameters are replaced as expected

After you assign values to the scheduling parameters, we recommend that you use the smoke testing feature to specify a data timestamp for the node and test whether the code of the node is executed and the scheduling parameters are replaced as expected when the node is scheduled to run. If the node is not run as expected, you can adjust the scheduling configurations of the node at the earliest opportunity to prevent the invalid configurations from affecting the running of the node.

After you modify the code of the node, you must click the icon and then the icon to save the modifications and commit the node to the development environment. Otherwise, you cannot perform smoke testing on the node in the development environment.

Note

Before you click the or icon in the toolbar to run a node, you must manually assign constants to the variables that you want to reference in the code of the node. In this case, you cannot check whether the configured scheduling parameters work as expected.
When you perform smoke testing in the development environment, you are charged for the generated test instances. For more information about instance pricing, see Billing of serverless resource groups.

Perform smoke testing
View smoke testing logs

View the configurations of scheduling parameters in Operation Center in the production environment

To prevent unexpected configurations of scheduling parameters from affecting the running of an auto triggered node, we recommend that you check the configurations of the scheduling parameters for the auto triggered node on the Auto Triggered Nodes page in Operation Center in the production environment after the auto triggered node is deployed. For more information about auto triggered nodes, see View and manage auto triggered nodes. 生产环境确认参数配置

Note

If the scheduling parameters of an auto triggered node are not configured as expected or you cannot find the auto triggered node in Operation Center, check whether the node is successfully deployed. For more information about how to deploy nodes, see Deploy nodes.

Example on how to configure scheduling parameters

This section describes how to perform smoke testing in the development environment to check whether the scheduling parameters of a node work as expected. In this section, an ODPS SQL node is used. After the node is deployed, you can view the configurations of the scheduling parameters in Operation Center in the production environment.

Note

For more information about how to configure scheduling parameters for different types of nodes, see Configure scheduling parameters for different types of nodes.

Edit the code of the node and configure the scheduling parameters.
The following figure shows the code and scheduling parameter configurations of the ODPS SQL node.
1. Define variables in the code.
  Define variables '${var1}' and '${var2}' to reference built-in parameters and define variables '${var3}' and '${var4}' to reference custom parameters in the code of the ODPS SQL node, as shown in area 1 in the preceding figure.
2. Assign values to the variables.
  Go to the Parameters section of the Properties tab and assign values to the variables, as shown in area 2 in the preceding figure.
  - var1=$bizdate: the data timestamp in the yyyymmdd format.
  - var2=$cyctime: the time when the node is scheduled to run in the yyyymmddhh24miss format.
  - var3=${yyyymmdd}: the data timestamp in the yyyymmdd format.
  - var4=$[yyyymmddhh24miss]: the time when the node is scheduled to run in the yyyymmddhh24miss format.
3. Optional: Configure the scheduling frequency.
  Configure the ODPS SQL node to run at an interval of a specified number of hours, as shown in area 3 in the preceding figure.
  Note
  You can determine whether to configure the scheduling frequency for a node based on your business requirements. In this example, the scheduling frequency is configured.
  - Start from: 16:00
  - End at: 23:59
  - Interval: 1 hour
  For information about how to configure the scheduling frequency, see Configure time properties.
In the top toolbar of the configuration tab of the node, click the icon and then the icon to save the configurations of the ODPS SQL node and commit the ODPS SQL node.
Perform smoke testing in the development environment.
1. Click the icon. In the Perform Smoke Testing in Development Environment dialog box, configure a data timestamp to simulate the scheduling frequency of the node.
  In this example, configure a data timestamp in the following way:
  - Data timestamp: 2022-03-09
  - Start from: 16:00
  - End at: 17:00
  The ODPS SQL node is scheduled by hour. Two instances are expected to be generated for the node at 16:00 and 17:00 on March 10, 2022.
  Note
  The data timestamp is one day earlier than the time when the node is scheduled to run. Therefore, the node is scheduled to run on March 10, 2022.
  The following results are expected for the instance that is generated at 16:00:
  - var1=20220309
  - var2=20220310160000
  - var3=20220309
  - var4=20220310160000
  The following results are expected for the instance that is generated at 17:00:
  - var1=20220309
  - var2=20220310170000
  - var3=20220309
  - var4=20220310170000
2. Click OK. The node is run at its scheduling time.
3. After the smoke testing is complete, click the icon to view the smoke testing logs.
  The two instances generated by the node are successfully run and the test result meets expectations.
On the configuration tab of the ODPS SQL node, click Deploy on the right side of the top navigation bar to deploy the current node.
For more information about how to deploy nodes, see Deploy nodes.
Go to Operation Center and check the configurations of the scheduling parameters of the node.
1. On the DataStudio page, click Operation Center on the right side of the top navigation bar to go to the Operation Center page.
2. In the left-side navigation pane, choose Auto Triggered Node O&M > Auto Triggered Nodes. On the Auto Triggered Nodes page, find the node that you want to manage.
  Note
  You can find a node on the Auto Triggered Nodes page only after the node is deployed.
3. Choose More > View Auto Triggered Task Details in the Actions column of the node. On the Node Information page, view the value of Execution Parameters.
  In this example, the scheduling parameters of the node are configured as expected in the following way: var1=$bizdate var2=$cyctime var3=${yyyymmdd} var4=$[yyyymmddhh24miss].