E-MapReduce:Edit jobs

Background information

Prerequisites

A project is created. For more information, see Manage projects.

Create a job

1. Go to the Data Platform tab.
   1. Log on to the Alibaba Cloud EMR console by using your Alibaba Cloud account.
   2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
   3. Click the Data Platform tab.
2. In the Projects section of the page that appears, find the project that you want to manage and click Edit Job in the Actions column.
3. Create a job.
   1. In the Edit Job pane on the left side of the page that appears, right-click the folder on which you want to perform operations and select Create Job.
      Note You can also right-click the folder and select Create Subfolder, Rename Folder, or Delete Folder to perform the corresponding operation.
   2. In the Create Job dialog box, specify Name and Description, and then select a specific job type from the Job Type drop-down list.

      E-MapReduce (EMR) supports the following types of jobs in data development: Shell, Hive, Hive SQL, Spark, Spark SQL, Spark Shell, Spark Streaming, MapReduce, Sqoop, Pig, Flink, Streaming SQL, Presto SQL, and Impala SQL.

      Note After the job is created, you cannot change the type of the job.
   3. Click OK.
      After a job is created, you can configure and edit the job.

Configure a job

For more information about how to develop and configure each type of job, see Jobs. This section describes how to configure the parameters of a job on the Basic Settings, Advanced Settings, Shared Libraries, and Alert Settings tabs in the Job Settings panel.

1. In the upper-right corner of the job page, click Job Settings.
2. In the Job Settings panel, configure the parameters on the Basic Settings tab.

   Section and parameter		Description
   Job Overview	Name	The name of the job.
   	Job Type	The type of the job.
   	Retries	The number of retries that are allowed if the job fails. The value of this parameter ranges from 0 to 5.
   	Actions on Failures	The action that you can perform if the job fails. Valid values: Pause: Suspend the current workflow if the job fails. Run Next Job: Continue to run the next job if the job fails. You can determine whether to turn on the Use Latest Job Content and Parameters switch based on your business requirements. If you turn off this switch, a job instance is generated based on the original job content and parameters after you rerun a job that fails. If you turn on this switch, a job instance is generated based on the latest job content and parameters after you rerun a job that fails.
   	Description	The description of the job. If you want to modify the description of the job, you can click Edit on the right side of this parameter.
   Resources		The resources that are required to run the job, such as JAR packages and user-defined functions (UDFs). Click the icon on the right side to add resources. Upload the resources to Object Storage Service (OSS) first. Then, you can add them to the job.
   Configuration Parameters		The variables that you want to reference in the job script. You can reference a variable in your job script in the format of ${Variable name}. Click the icon on the right side to add a variable in the key-value pair format. You can determine whether to select Password to hide the value based on your business requirements. The key indicates the name of the variable. The value indicates the value of the variable. In addition, you can configure a time variable based on the start time of scheduling. For more information, see Configure job time and date.

3. Click the Advanced Settings tab and configure the parameters.

   Section	Parameter and description
   Mode	Job Submission Node: the mode to submit the job. For more information, see Job submission modes. Valid values: Worker Node: The job is submitted to YARN by using a launcher, and YARN allocates resources to run the job. Header/Gateway Node: The job runs as a process on the allocated node. Estimated Maximum Duration: the estimated maximum running duration of the job. Valid values: 0 to 10800. Unit: seconds.
   Environment Variables	The environment variables that are used to run the job. You can also export environment variables from the job script. Example 1: Configure a Shell job with the code echo ${ENV_ABC}. If you set the ENV_ABC variable to 12345, a value of 12345 is returned after you run the echo command. Example 2: Configure a Shell job with the code java -jar abc.jar. Content of the abc.jar package: public static void main(String[] args) {System.out.println(System.getEnv("ENV_ABC"));} If you set the ENV_ABC variable to 12345, a value of 12345 is returned after you run the job. The effect of setting the ENV_ABC variable in the Environment Variables section is equivalent to running the following script: export ENV_ABC=12345 java -jar abc.jar
   Scheduling Parameters	The parameters used to schedule the job, including Queue, Memory (MB), vCores, Priority, and Run By. If you do not configure these parameters, the default settings of the Hadoop cluster are used. Note The Memory (MB) parameter specifies the memory quota for the launcher.

4. Click the Shared Libraries tab.
   In the Dependent Libraries section, specify Libraries.

   Job execution depends on some library files related to data sources. EMR publishes the libraries to the repository of the scheduling center as dependency libraries. You must specify dependency libraries when you create a job. To specify a dependency library, enter its reference string, such as sharedlibs:streamingsql:datasources-bundle:2.0.0.

5. Click the Alert Settings tab and configure the alert parameters.

   Parameter	Description
   Execution Failed	Specifies whether to send a notification to an alert contact group or a DingTalk alert group if the job fails.
   Action on Startup Timeout	Specifies whether to send a notification to an alert contact group or a DingTalk alert group if the job startup times out.
   Job execution timed out.	Specifies whether to send a notification to an alert contact group or a DingTalk alert group if the job execution times out.

Add annotations

!!! @<Annotation name>: <Annotation content>

!!! @rem: This is a comment.

!!! @env: ENV_1=ABC

!!! @var: var1="value1 and \"one string end with 3 spaces\"   "
!!! @var: var2=${yyyy-MM-dd}

!!! @resource: oss://bucket1/dir1/file.jar

!!! @sharedlibs: sharedlibs:streamingsql:datasources-bundle:1.7.0,...

!!! @scheduler.queue: default

!!! @scheduler.vmem: 1024

!!! @scheduler.vcores: 1

!!! @scheduler.priority: 1

!!! @scheduler.user: root

Run a job

1. Run the job that you created.
   1. On the job page, click Run in the upper-right corner to run the job.
   2. In the Run Job dialog box, select a resource group and the cluster that you created.
   3. Click OK.
2. View running details.
   1. Click the Log tab in the lower part of the job page to view the operational logs.
      
   2. Click the Records tab to view the execution records of the job instance.
   3. Click Details in the Action column of a job instance to go to the Scheduling Center tab. On this tab, you can view the details about the job instance.

Operations that you can perform on jobs

Job submission modes

The spark-submit process, which is the launcher in a data development module, is used to submit Spark jobs. In most cases, this process occupies more than 600 MiB of memory. The Memory (MB) parameter in the Job Settings panel specifies the size of the memory allocated to the launcher.

Memory consumed by a job instance = Memory consumed by the launcher + Memory consumed by a job that corresponds to the job instance

Memory consumed by a job = Memory consumed by the spark-submit logical module (not the process) + Memory consumed by the driver + Memory consumed by the executor