All Products
Search
Document Center

E-MapReduce:Parameters

Last Updated:Aug 14, 2023

E-MapReduce (EMR) Workflow provides built-in time parameters, task parameters (local parameters), workflow parameters (global parameters), and project parameters. EMR Workflow also supports the parameter passing feature.

Configure parameters

Built-in time parameters

Define parameters

You do not need to define built-in time parameters. You can directly use them in tasks.

Use parameters

EMR Workflow supports the built-in time parameters in the $[...] format. A built-in time parameter in the $[yyyyMMddHHmmss] format can be split and recombined into multiple formats, such as $[yyyyMMdd], $[HHmmss], and $[yyyy-MM-dd]. You can use one of the following methods to modify a built-in time parameter:

  • Use the add_months() function. This function is used to add or subtract months. In the following examples, the first parameter [yyyyMMdd] in the parenthesis indicates the format of the returned time. The second parameter in the parenthesis indicates the number of months to be added or subtracted.

    • Add N years: $[add_months(yyyyMMdd,12*N)]

    • Subtract N years: $[add_months(yyyyMMdd,-12*N)]

    • Add N months: $[add_months(yyyyMMdd,N)]

    • Subtract N months: $[add_months(yyyyMMdd,-N)]

  • Add +Number or -Number to a custom time parameter.

    • Add N weeks: $[yyyyMMdd+7*N]

    • Subtract N weeks: $[yyyyMMdd-7*N]

    • Add N days: $[yyyyMMdd+N]

    • Subtract N days: $[yyyyMMdd-N]

    • Add N hours: $[HHmmss+N/24]

    • Subtract N hours: $[HHmmss-N/24]

    • Add N minutes: $[HHmmss+N/24/60]

    • Subtract N minutes: $[HHmmss-N/24/60]

Task parameters

Task parameters, which are also known as local parameters, are valid for nodes. By default, the custom parameters that are defined when you create a node apply only to the current node. If you specify that the parameters can be passed to the descendant nodes of the current node, the parameters can be applied to the descendant nodes of the current node.

Define parameters

In the Current node settings dialog box, click the image.png icon below Custom Parameters to add one or more custom parameters. The following figure shows how to define a custom parameter.image.png

Custom parameter defined in this example:

  • dt: the name of the custom parameter.

  • IN: specifies that the parameter applies only to the current node. A value of OUT specifies that the parameter can be passed to the descendant nodes of the current node.

  • VARCHAR: the data type of the parameter.

  • $[yyyy-MM-dd]: the value of the parameter.

Use parameters

When you create a node, specify local parameters in the ${...} format.

Workflow parameters

Workflow parameters, which are also known as global parameters, are valid for all nodes of a workflow and are configured when you save the workflow.

Define parameters

When you save a workflow, you can add one or more global parameters and specify the names and values of the parameters.

image.png

Use parameters

When you save a workflow, specify global parameters in the ${...} format.

Project parameters

Project parameters are valid for all workflows in a project.

Define parameters

To define a project parameter, perform the following operations: On the details page of a project, choose Project Setting > Project Parameter in the left-side navigation pane. On the Project Parameter page, click Create Parameter. In the Create Parameter dialog box, configure the Parameter Name and Parameter Value parameters.

image.png

Use parameters

When you create a node, specify project parameters in the ${...} format.

Pass parameters

EMR Workflow allows you to pass parameters between nodes. You can pass the parameters of a node to the descendant nodes of the node.

When you define a custom parameter for a node, select OUT to specify that the custom parameter can be passed to the descendant nodes of the current node. In addition, you must enter an echo statement in the script for the current node. The echo statement is in the format of echo "#{setValue(key=value)}". In this statement, key indicates the name of the custom parameter of the OUT type. value indicates the value that you want to pass to the descendant nodes of the current node, which can be a constant, a value of another task parameter, or a value of a variable in the bash script. To use the parameters that are passed from a node to its descendant nodes, you need to only specify the parameters in the ${...} format for the descendant nodes.

Examples:

Pass a constant

When you create a node, define a custom parameter named res of the OUT type and enter echo "#{setValue(res=1234)}" in the script. This way, you can pass the res parameter whose value is 1234 to the descendant nodes of the current node.

image.png

Pass a task parameter

When you create a node, define a custom parameter named param of the IN type and set its value to 123. Then, define another custom parameter named res of the OUT type and enter echo "#{setValue(res=${param})}" in the script. This way, you can use the res parameter to pass the param parameter whose value is 123 to the descendant nodes of the current node.

image.png

Pass a variable in the bash script

When you create a node, define a custom parameter named res of the OUT type and enter echo "#{setValue(res=${RESULT})}" in the script. This way, you can use the res parameter to pass the RESULT variable whose value is 123 to the descendant nodes of the current node.

image.png

Parameter priority

When you use EMR Workflow, the following types of parameters are involved:

If multiple parameters of different types have the same name, the specified parameter values are applied based on the following priority: Task parameters (local parameters) > Parameters passed to the descendant nodes of a node > Workflow parameters (global parameters) > Project parameters.