DataWorks:Create an EMR Spark node

A JAR package is less than 200 MB in size

1. Create an EMR JAR resource.

   You can upload the JAR package from your on-premises machine to the DataWorks console as an EMR JAR resource. This way, you can manage the JAR package in the DataWorks console in a visualized manner. After you create an EMR JAR resource, you must commit the resource. For more information, see Create and use an EMR resource.

   

   Note

   The first time you create an EMR JAR resource, you must perform authorization as prompted first if you want the JAR package to be stored in OSS after the JAR package is uploaded.

2. Reference the EMR JAR resource.

   1. Double-click the name of the created EMR Spark node to go to the configuration tab of the node.

   2. Find the desired EMR JAR resource under Resource in the EMR folder, right-click the resource name, and then select Insert Resource Path.

   3. Resource reference code is automatically added to the configuration tab of the EMR Spark node. Sample code:

      ##@resource_reference{"spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar"}
      spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar

      If the automatic addition of the preceding code is successful, the resource is referenced. spark-examples_2.12-1.0.0-SNAPSHOT-shaded.jar is the name of the JAR package that you uploaded.

   4. Rewrite the code of the EMR Spark node and add the spark-submit command. The following sample code is only for reference.

      Note

      You cannot add comments when you write code for an EMR Spark node. If you add comments, an error is reported when you run the EMR Spark node. You can refer to the following sample code to rewrite the code of an EMR Spark node.

      ##@resource_reference{"spark-examples_2.11-2.4.0.jar"}
      spark-submit --class org.apache.spark.examples.SparkPi --master yarn  spark-examples_2.11-2.4.0.jar 100

      Parameter description:

      • org.apache.spark.examples.SparkPi: the main class of the task in the compiled JAR package.

      • spark-examples_2.11-2.4.0.jar: the name of the JAR package that you uploaded.

      • You can keep the settings of other parameters unchanged. You can also run the following command to view the help documentation for using the spark-submit command and modify the spark-submit command based on your business requirements.

        Note

        • If you want to use a parameter that is simplified by running the spark-submit command, such as --executor-memory 2G, in an EMR Spark node, you need to add the parameter to the code of the EMR Spark node.

        • You can use Spark nodes on YARN to submit jobs only if your nodes are in cluster mode.

        • If you commit a node by using spark-submit, we recommend that you set deploy-mode to cluster rather than client.

        spark-submit --help

        

A JAR package is greater than or equal to 200 MB in size

1. Store the JAR package in HDFS of EMR.

   You cannot upload the JAR package from your on-premises machine to the DataWorks console as a DataWorks resource. We recommend that you store the JAR package in HDFS of EMR and record the storage path of the JAR package. This way, you can reference the JAR package in this path when you use DataWorks to schedule Spark tasks.

2. Reference the JAR package.

   You can reference the JAR package by specifying the storage path of the JAR package in the code of an EMR Spark node.

   1. Double-click the name of the created EMR Spark node to go to the configuration tab of the node.

   2. Write the spark-submit command. Example:

      spark-submit --master yarn
      --deploy-mode cluster
      --name SparkPi
      --driver-memory 4G
      --driver-cores 1
      --num-executors 5
      --executor-memory 4G
      --executor-cores 1
      --class org.apache.spark.examples.JavaSparkPi
      hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar 100

      Parameter description:

      • hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar: the storage path of the JAR package in HDFS.

      • org.apache.spark.examples.JavaSparkPi: the main class of the task in the compiled JAR package.

      • Other parameters are configured in the EMR cluster that is used. You can modify the parameters based on your business requirements. You can also run the following command to view the help documentation for using the spark-submit command and modify the spark-submit command based on your business requirements.

        Important

        • If you want to use a parameter that is simplified by running the spark-submit command, such as --executor-memory 2G, in an EMR Spark node, you need to add the parameter to the code of the EMR Spark node.

        • You can use Spark nodes on YARN to submit jobs only if your nodes are in cluster mode.

        • If you commit a node by using spark-submit, we recommend that you set deploy-mode to cluster rather than client.

        spark-submit --help

        

DataLake cluster or custom cluster: created on the EMR on ECS page

EMR Serverless Spark cluster

For more information about parameter settings, see the Step 3: Submit a Spark job section of the "Use the spark-submit CLI to submit a Spark job" topic.

Spark cluster: created on the EMR on ACK page

Hadoop cluster: created on the EMR on ECS page