DataWorks:Create an EMR Spark Streaming node

Create and reference an EMR JAR resource

If you use an EMR DataLake cluster, you can perform the following steps to reference an EMR JAR resource:

1. Prepare the EMR JAR sample code.

   Plaintext

   spark-submit --master yarn
   --deploy-mode cluster
   --name SparkPi
   --driver-memory 4G
   --driver-cores 1
   --num-executors 5
   --executor-memory 4G
   --executor-cores 1
   --class org.apache.spark.examples.JavaSparkPi
   hdfs:///tmp/jars/spark-examples_2.11-2.4.8.jar 100

   Note

   If an EMR Spark Streaming node depends on large amounts of resources, the resources cannot be uploaded by using the DataWorks console. In this case, you can store the resources in Hadoop Distributed File System (HDFS) and then reference the resources in the code of the EMR Spark Streaming node. Sample code:

2. Create an EMR JAR resource. For more information, see Create and use an EMR resource. The first time you use an EMR JAR resource, click Authorize to authorize DataWorks to access the EMR JAR resource.

3. Reference the EMR JAR resource.

   1. Open the EMR Spark Streaming node. The configuration tab of the node appears.

   2. Find the resource that you want to reference below Resource in the EMR folder, right-click the resource name, and then select Insert Resource Path.

   3. If the clause that is in the ##@resource_reference{""} format appears on the configuration tab of the EMR Spark Streaming node, the resource is referenced. Then, run the following code. You must replace the information in the following code with the actual information. The information includes the resource package name, bucket name, and directory.

      ##@resource_reference{"examples-1.2.0-shaded.jar"}
      --master yarn-cluster --executor-cores 2 --executor-memory 2g --driver-memory 1g --num-executors 2 --class com.aliyun.emr.example.spark.streaming.JavaLoghubWordCount examples-1.2.0-shaded.jar <logService-project> <logService-store> <group> <endpoint> <access-key-id> <access-key-secret>

Develop SQL code

On the configuration tab of the EMR Spark Streaming node, write code for the node. Sample code:

spark-submit --master yarn-cluster --executor-cores 2 --executor-memory 2g --driver-memory 1g --num-executors 2 --class com.aliyun.emr.example.spark.streaming.JavaLoghubWordCount examples-1.2.0-shaded.jar <logService-project> <logService-store> <group> <endpoint> <access-key-id> <access-key-secret>

(Optional) Configure advanced parameters

You can configure advanced parameters on the Advanced Settings tab of the configuration tab of the current node. For more information about how to configure the parameters, see Spark Configuration. The following table describes the advanced parameters that can be configured.

Run the Spark Streaming task

1. In the toolbar, click the icon. In the Parameters dialog box, select the desired resource group from the Resource Group Name drop-down list and click Run.

   Note

   • If you want to access a data source over the Internet or a virtual private cloud (VPC), you must use the resource group for scheduling that is connected to the data source. For more information, see Network connectivity solutions.

   • If you want to change the resource group in subsequent operations, you can click the (Run with Parameters) icon to change the resource group in the Parameters dialog box.

2. Click the icon in the top toolbar to save the SQL statements.

3. Optional. Perform smoke testing.

   You can perform smoke testing on the node in the development environment when you commit the node or after you commit the node. For more information, see Perform smoke testing.