Supported formats of scheduling parameters

In DataWorks, a node is scheduled to run based on the scheduling parameters that you configure for the node. The values of the scheduling parameters are automatically replaced based on the value assignment rule for the scheduling parameters each time the node is run. This way, the values of the scheduling parameters are dynamically replaced at the scheduling time of the node. This topic describes the supported formats of scheduling parameters and how to use scheduling parameters.

Background information

After you define variables in the ${Variable} format in the code of a node and configure the scheduling parameters that are assigned to the variables as values in the Parameters section of the Properties tab, the values of the scheduling parameters are dynamically replaced in the node code when the node is scheduled to run. The format of a scheduling parameter varies based on the type of a node. For example, you can configure scheduling parameters for a Shell node only in the $N format. N specifies an integer that starts from 1. For more information, see Configure scheduling parameters for different types of nodes.

Note

You can assign a constant to a scheduling parameter. If you assign a constant to a scheduling parameter for a node, the same value is passed to the scheduling parameter each time the node is run. This topic describes how the values of scheduling parameters are dynamically replaced in the code of a node based on the configurations of the scheduling parameters.

This topic consists of the following sections:

Concepts related to scheduling parameters
Supported formats of scheduling parameters
Precautions for using scheduling parameters
Appendix: Parameters
Appendix: Relationships between scheduling parameters and the data timestamp, scheduling time, and actual running time of a node
Appendix: Description of daylight saving time

After you understand the supported formats of scheduling parameters, you can configure and use scheduling parameters based on your business requirements. For more information, see What to do next: Configure and use scheduling parameters.

Concepts

In most cases, scheduling parameters are used in the scenarios in which you want to specify a dynamic point in time. In the scenarios, the value of a scheduling parameter is determined based on the data timestamp or scheduling time of a node. You must understand the concepts of the data timestamp and scheduling time of a node before you configure scheduling parameters. This helps you configure the value formats for the scheduling parameters. The following table describes the concepts.

Concept

Description

Data timestamp

The previous day of the scheduling time (the time when you want to schedule the node). In offline computing scenarios, a data timestamp represents the date on which a business transaction is conducted. The value of a data timestamp is accurate to the day.

For example, if you collect statistical data on the turnover of the previous day on the current day, the previous day is the date on which the business transaction is conducted and represents the data timestamp.

Scheduling time

The time when you want to schedule the node to process business data. The scheduling time is accurate to the second.

The scheduling time can be different from the actual time at which the node is scheduled to run. The actual time at which a node is run is affected by multiple factors.

In most cases, scheduling parameters are used to obtain different time data. You can choose to use the data timestamp or scheduling time of a node to obtain time data based on your business requirements.

Note

DataWorks provides some built-in variables that have fixed formats. DataWorks also allows you to use the methods that are described in the following table to customize time formats.

Method to obtain time data

Parameter format

Sample parameter

References

Obtain time data based on the data timestamp of a node

In most cases, a custom time parameter that uses the data timestamp of a node to obtain time data consists of ${...}, yyyy, yy, mm, and dd. You can use a ${...} parameter to specify a time that is N days, weeks, months, or years earlier or later than the current data timestamp.

Note

The data timestamp of a node is calculated by subtracting one day from the date of the scheduling time. This indicates that the value of the ${yyyymmdd} parameter is the same as that of the $[yyyymmdd-1] parameter.

You can use a custom time parameter in the ${...} format, such as ${yyyymmdd} or ${yyyy-mm-dd} to obtain time data. Examples:

${yyyymmdd±N}
${yyyymmdd±7*N}
${yy±N}
${mm}
${yyyy-mm-dd±N}

For more examples of custom time parameters in the ${...} format, see Custom parameters in the ${...} format.

Obtain time data based on the scheduling time of a node

In most cases, a custom time parameter that uses the scheduling time of a node to obtain time data consists of $[...], yyyy, yy, mm, dd, hh24, mi, and ss. You can use a $[...] parameter to specify a time that is N days, weeks, months, years, hours, or minutes earlier or later than the current scheduling time.

You can use a custom time parameter in the $[...] format, such as $[yyyymmddhh24miss], to obtain time data. For example, you can use the $[yyyymmdd-1-1/24] parameter to specify a time that is 1 hour earlier than the current time on the previous day.

For more examples of custom time parameters in the $[...] format, see Custom parameters in the $[...] format.
If you want to specify a time that is accurate to the hour or minute, cross-day issues may occur. For information about solutions to the issues, see I run an instance of a node at 00:00 on the current day to analyze the data in the partition that corresponds to 23:00 on the previous day. However, the data in the partition that corresponds to 23:00 on the current day is analyzed. What do I do?.

If you cannot use the preceding custom time parameters to obtain the desired time data, we recommend that you use the preceding custom time parameters to obtain data and then process the data by performing the operations that are described in the following table to obtain the desired time data.

Sample scenario	Description	References
Call engine functions to convert time data into timestamps.	Call engine functions in the code of a node to obtain the desired time data.	Process the return values of scheduling parameters
${mm}01: Obtain the first day of each month.	Concatenate strings to obtain the desired time data.	None

Precautions

${...} parameters
If you want to specify a time that is N years or months ago, use a ${...} parameter.
- ${...} parameters are accurate to the day. You cannot use a ${...} parameter to specify a time that is accurate to the hour, minute, or second. For example, you cannot specify a ${...} parameter in the ${yyyy-mm-dd-1/24} format.
- If you want to specify a calculation-based time, we recommend that you use a ${...} parameter. For example, you can specify a parameter in the ${yyyy-N} or ${mm-N} format.
$[…] parameters
- If you want to specify a time that is N hours or minutes ago, use a $[...] parameter.
  - $[…] parameters can be used to specify a time that is accurate to the hour, minute, or second. You cannot use the $[yyyy-N] or $[mm-N] parameter to obtain a time that is N years or months ago. To specify a time that is in a specific year or month, use ${...} parameters.
  - If you want to specify a time that is accurate to the second, we recommend that you use a $[...] parameter. Example: $[yyyy-mm-dd-1-1/24].
- The value of a $[...] parameter does not change with the actual time at which the instance generated for a node is run.
  The value of a $[...] parameter is determined by the scheduling time of the instance generated for a node. The value is determined when the instance is generated. Therefore, the value of a $[...] parameter does not change with the actual time at which the instance generated for a node is run. For more information, see Appendix: Relationships between scheduling parameters and the data timestamp, scheduling time, and actual running time of a node.
Note
- When you use a $[...] parameter to specify a time that is accurate to the hour, both the 24-hour clock format and the 12-hour clock format are supported.
- hh24 indicates a 24-hour clock. hh indicates a 12-hour clock.
${bizdate} variable
The code of some nodes contains the ${bizdate} variable. This variable has no special meaning. You can customize variable names in the code of a node based on your business requirements.
Solutions to the cross-day issues that may occur when you specify a time that is accurate to the hour or minute
If you want to specify a time that is accurate to the hour or minute, cross-day issues may occur. For information about solutions to the issues, see I run an instance of a node at 00:00 on the current day to analyze the data in the partition that corresponds to 23:00 on the previous day. However, the data in the partition that corresponds to 23:00 on the current day is analyzed. What do I do?.

Appendix: Parameters

For information about the differences between custom time parameters in the ${...} format and those in the $[...] format, see Differences between ${...} and $[...] parameters.

Custom parameters in the ${...} format

The following table describes the sample common time formats.

Interval that you want to add or subtract	Custom parameter
N years ago or later	`${yyyy±N}`
N months ago or later	`${yyyymm±N}`
N weeks ago or later	`${yyyymmdd±7*N}`
N days ago or later	`${yyyymmdd±N}`
N days ago or later	`${yyyymmdd±N}`
N years ago or later in the yyyy format	`${yyyy±N}`
N years ago or later in the yy format	`${yy±N}`

Important

To specify a time that is N months or N years ago, we recommend that you use ${...} parameters.
You can use ${…} parameters to add or subtract only N days, months, or years.

Custom parameters in the $[...] format

The following table describes the sample common time formats.

Interval that you want to add or subtract	Custom parameter
N years later	`$[add_months(yyyymmdd,12*N)]`
N years ago	`$[add_months(yyyymmdd,-12*N)]`
N months later	`$[add_months(yyyymmdd,N)]`
N months ago	`$[add_months(yyyymmdd,-N)]`
N weeks ago or later	`$[yyyymmdd±7*N]`
N days ago or later	`$[yyyymmdd±N]`
N hours ago or later	You can use one of the following methods to add or subtract N hours: `$[hh24miss±N/24]` and `$[hh24±N/24]` `$[Custom time format±N/24]` For example, you can use the following custom time parameters to specify a time that is 1 hour ago: A specific month: `$[mm-1/24]` A specific year: `$[yyyy-1/24]` A specific month of a specific year: `$[yyyymm-1/24]` A specific day of a specific month of a specific year: `$[yyyymmdd-1/24]` The day before a specific day: `$[yyyymmdd-1-1/24]`
N minutes ago or later	You can use one of the following methods to add or subtract N minutes: `$[hh24miss±N/24/60]` `$[yyyymmddhh24miss±N/24/60]` `$[mi±N/24/60]` `$[Custom time format±N/24/60]` For example, you can use the following custom time parameters to specify a time that is 15 minutes earlier than the scheduling time of a node: `$[yyyy-15/24/60]` `$[yyyymm-15/24/60]` `$[yyyymmdd-15/24/60]` `$[hh24-15/24/60]` `$[mi-15/24/60]`

hh24 (24-hour format) and hh (12-hour format) indicate the hour. The value of the $cyctime parameter is accurate to the hour, minute, or second. Therefore, the value of a $[...] parameter is accurate to the hour, minute, or second. You can use engine functions to specify custom parameters in the formats that are not described in this topic. For more information, see Process the return values of scheduling parameters.

Important

If you want to specify a time that is N hours or N minutes ago, use a $[...] parameter.
The value of a scheduling parameter is determined when an instance of a node is generated and does not change with the actual time at which the node instance is run.

Built-in parameters

DataWorks supports the built-in parameters that are described in the following table. The built-in parameters can be used as values for the scheduling parameters. However, the built-in parameters are not flexible. We recommend that you use custom time parameters as values for the scheduling parameters.

Built-in parameter	Description
$bizdate	The data timestamp of the node. The value of this parameter is in the `yyyymmdd` format. The value of this parameter is the same as that of the `${yyyymmdd}` custom time parameter. This parameter is widely used. By default, the data timestamp of a node is one day earlier than the scheduling time of the node.
$cyctime	The scheduling time of the node. The value of this parameter is in the `yyyymmddhh24miss` format. The value of this parameter is the same as that of the `$[yyyymmddhh24miss]` custom time parameter.
$gmtdate	The current date. The value of this parameter is in the `yyyymmdd` format. By default, the value of this parameter is the current date. When you backfill data for a node, set this parameter to a time that is `one day later than the data timestamp of the node`.
$bizmonth	The month indicated by the data timestamp of the node. The value of this parameter is in the `yyyymm` format. If the month indicated by the data timestamp of the node is the current month, the value of this parameter is `one month earlier than the month indicated by the data timestamp of the node`. If the month indicated by the data timestamp of the node is not the current month, the value of this parameter is `the month indicated by the data timestamp of the node`.
$jobid	The ID of the workflow to which the node belongs.
$nodeid	The ID of the node.
$taskid	The ID of the instance generated for the node.

Appendix: Relationships between scheduling parameters and the data timestamp, scheduling time, and actual running time of a node

Relationship	Description and example
Relationship between scheduling parameters and the actual running time of a node	The value of a scheduling parameter is determined when an instance of a node is generated and does not change with the actual time at which the node instance is run.
Relationship between scheduling parameters and the scheduling time of a node	If you use a $[...] parameter to specify a time that is accurate to the hour or minute, the value of the $[...] parameter is determined based on the scheduling time, which is the time at which the instance generated for the node is scheduled to run. Value replacement if the node is scheduled to run every hour from `00:00` to `23:59`: If the first instance generated for the node is scheduled to run at `00:00`, the value of hh24 or hh is replaced with `00`. If the second instance generated for the node is scheduled to run at `01:00`, the value of hh24 or hh is replaced with `01`.
Relationships between scheduling parameters and the data timestamp and scheduling time of a node

Appendix: Description of daylight saving time

In DataWorks, nodes can be run as expected on the day when daylight saving time begins or ends.