DataWorks:Create an EMR Spark node

The current node can reference an OSS resource by using the OSS REF method. When you run a task on the node, DataWorks automatically loads the OSS resource specified in the node code. This method is commonly used in scenarios in which JAR dependencies are required in EMR tasks or EMR tasks need to depend on scripts.

1. Develop a JAR package.

   1. Prepare code dependencies.

      You can access the EMR cluster and view required code dependencies in the /usr/lib/emr/spark-current/jars/ path of the master node. The following information uses Spark 3.4.2 as an example. You must open an existing IntelliJ IDEA project, and add Project Object Model (POM) dependencies and reference plug-ins.

      Add POM dependencies

      <dependencies>
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-core_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
              <!-- Apache Spark SQL -->
              <dependency>
                  <groupId>org.apache.spark</groupId>
                  <artifactId>spark-sql_2.12</artifactId>
                  <version>3.4.2</version>
              </dependency>
      </dependencies>

      Reference plug-ins

      <build>
              <sourceDirectory>src/main/scala</sourceDirectory>
              <testSourceDirectory>src/test/scala</testSourceDirectory>
              <plugins>
                  <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-compiler-plugin</artifactId>
                      <version>3.7.0</version>
                      <configuration>
                          <source>1.8</source>
                          <target>1.8</target>
                      </configuration>
                  </plugin>
                  <plugin>
                      <artifactId>maven-assembly-plugin</artifactId>
                      <configuration>
                          <descriptorRefs>
                              <descriptorRef>jar-with-dependencies</descriptorRef>
                          </descriptorRefs>
                      </configuration>
                      <executions>
                          <execution>
                              <id>make-assembly</id>
                              <phase>package</phase>
                              <goals>
                                  <goal>single</goal>
                              </goals>
                          </execution>
                      </executions>
                  </plugin>
                  <plugin>
                      <groupId>net.alchim31.maven</groupId>
                      <artifactId>scala-maven-plugin</artifactId>
                      <version>3.2.2</version>
                      <configuration>
                          <recompileMode>incremental</recompileMode>
                      </configuration>
                      <executions>
                          <execution>
                              <goals>
                                  <goal>compile</goal>
                                  <goal>testCompile</goal>
                              </goals>
                              <configuration>
                                  <args>
                                      <arg>-dependencyfile</arg>
                                      <arg>${project.build.directory}/.scala_dependencies</arg>
                                  </args>
                              </configuration>
                          </execution>
                      </executions>
                  </plugin>
              </plugins>
          </build>

   2. Write code. Sample code:

      package com.aliyun.emr.example.spark
      
      import org.apache.spark.sql.SparkSession
      
      object SparkMaxComputeDemo {
        def main(args: Array[String]): Unit = {
          // Create a Spark session.
          val spark = SparkSession.builder()
            .appName("HelloDataWorks")
            .getOrCreate()
      
          // Display the Spark version.
          println(s"Spark version: ${spark.version}")
        }
      }

   3. Package the code into a JAR file.

      After you write and save the preceding code, package the code into a JAR file. In this example, a file named SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar is generated.

2. Upload the JAR file.

   1. Log on to the OSS console. In the top navigation bar, select a desired region. Then, in the left-side navigation pane, click Buckets.

   2. On the Buckets page, find the desired bucket and click the bucket name to go to the Objects page.

      In this example, the onaliyun-bucket-2 bucket is used.

   3. On the Objects page, click Create Directory to create a directory that is used to store the JAR file.

      In the Create Directory panel, set Directory Name to emr/jars and click OK.

   4. Upload the JAR file to the created directory.

      Go to the created directory. Click Upload Object. In the Files to Upload section, click Select Files and add the SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar file. Then, click Upload Object.

3. Reference the JAR file.

   1. Write code that is used to reference the JAR file.

      On the configuration tab of the EMR Spark node, write code that is used to reference the JAR file.

      spark-submit --class com.aliyun.emr.example.spark.SparkMaxComputeDemo --master yarn ossref://onaliyun-bucket-2/emr/jars/SparkWorkOSS-1.0-SNAPSHOT-jar-with-dependencies.jar

      Parameter description:

      Parameter	Description
      class	The full name of the main class to be executed.
      master	The running mode of the Spark application.
      ossref file path	Format: ossref://{endpoint}/{bucket}/{object} endpoint: the endpoint of OSS. If the endpoint parameter is left empty, only a resource in an OSS bucket that resides in the same region as the current EMR cluster can be referenced. bucket: a container that is used to store objects in OSS. Each bucket has a unique name. You can log on to the OSS console to view all buckets within the current logon account. object: a file name or path that is stored in a bucket.

   2. Run a task on the EMR Spark node.

      After you write code, click the icon and select a created serverless resource group to run a task on the EMR Spark node. After the task finishes running, record application IDs that are displayed in the console, such as application_1730367929285_xxxx.

   3. View results.

      Create an EMR Shell node and run the yarn logs -applicationId application_1730367929285_xxxx command on the EMR Shell node to view running results.