E-MapReduce:Configure network connectivity between EMR Serverless Spark and a data source across VPCs

Prerequisites

A data source is prepared. In this example, a DataLake cluster that contains the Hive service is created. Built-in MySQL is selected for Metadata when the cluster is created. For more information, see Create a cluster.

Limits

• Only one network connection can be created.

• You can use only vSwitches in specific zones to create network connections. The following table lists the zones.

  Region name	Region ID	Zone name
  Germany (Frankfurt)	eu-central-1	Frankfurt Zone A Frankfurt Zone B
  Indonesia (Jakarta)	ap-southeast-5	Jakarta Zone A

Step 1: Create a network connection

1. Go to the Network Connection page.

   1. Log on to the EMR console.

   2. In the left-side navigation pane, choose EMR Serverless > Spark.

   3. On the Spark page, find the desired workspace and click the name of the workspace.

   4. In the left-side navigation pane of the EMR Serverless Spark page, choose Admin > Network Connection.

2. On the Network Connection page, click Create Network Connection.

3. In the Create Network Connection dialog box, configure parameters and click OK. The following table describes the parameters.

   Parameter	Description
   Name	The name of the network connection.
   VPC	The VPC in which your EMR cluster resides. If no VPC is available, click Create VPC to create a VPC in the VPC console. For more information, see Create and manage a VPC.
   vSwitch	The same vSwitch to which the EMR cluster is connected. If no vSwitch is available in the current zone, click vSwitch to create a vSwitch in the VPC console. For more information, see Create and manage a vSwitch. Important You can select only a vSwitch in specific zones. For more information, see the Limits section in this topic.

   If Succeeded is displayed in the Status column of the connection, the network connection is created.

   

Step 2: Configure security group rules for the EMR cluster

1. Obtain the CIDR block of the vSwitch that you specified when you created the network connection.

   You can log on to the VPC console and obtain the CIDR block of the desired vSwitch on the vSwitch page.

   

2. Configure security group rules.

   1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.

   2. On the EMR on ECS page, find the desired cluster and click the name of the cluster.

   3. In the Security section of the Basic Information tab, click the link to the right of Cluster Security Group.

   4. On the Security Group Details tab of the page that appears, click Add Rule, configure the Port Range and Authorization Object parameters, and then click Save in the Actions column.

      Parameter	Description
      Port Range	The port number. Enter 9083.
      Authorization Object	The CIDR block of the vSwitch obtained in the previous step. Important To prevent attacks from external users, we recommend that you do not set the Authorization Object parameter to 0.0.0.0/0.

(Optional) Step 3: Connect to Hive and query data in a Hive table

If you have created and configured a Hive table, skip this step.

1. Log on to the master node of the DataLake cluster in SSH mode. For more information, see Log on to a cluster.

2. Run the following command to open the Hive CLI:

   hive

3. Execute the following statement to create a table:

   CREATE TABLE my_table (id INT,name STRING);

4. Execute the following statements to insert data into the table:

   INSERT INTO my_table VALUES (1, 'John'); 
   INSERT INTO my_table VALUES (2, 'Jane');

5. Execute the following statement to query data in the table:

   SELECT * FROM my_table;

(Optional) Step 4: Prepare and upload a resource file

If you want to use a JAR job, you must prepare a resource file in advance. If you use a Spark SQL job, you can skip this step.

1. Create an on-premises Maven project.

   Sample code of the project

   package com.example;
   
   import org.apache.spark.sql.Dataset;
   import org.apache.spark.sql.Row;
   import org.apache.spark.sql.SparkSession;
   
   public class DataFrameExample {
       public static void main(String[] args) {
           // Create a Spark session. 
           SparkSession spark = SparkSession.builder()
                   .appName("HMSQueryExample")
                   .enableHiveSupport()
                   .getOrCreate();
   
           // Execute a query statement. 
           Dataset<Row> result = spark.sql("SELECT * FROM default.my_table");
   
           // Print the query results. 
           result.show();
   
           // Close the Spark session. 
           spark.stop();
       }
   }

   Sample code of the POM file

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
   
       <groupId>org.example</groupId>
       <artifactId>sparkDataFrame</artifactId>
       <version>1.0-SNAPSHOT</version>
   
       <properties>
           <maven.compiler.source>8</maven.compiler.source>
           <maven.compiler.target>8</maven.compiler.target>
           <spark.version>3.3.1</spark.version>
           <scala.binary.version>2.12</scala.binary.version>
       </properties>
   
       <dependencies>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-core_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-sql_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-hive_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
       </dependencies>
   </project>

2. Run the mvn package command to compile and package the files into the sparkDataFrame-1.0-SNAPSHOT.jar package.

3. In the left-side navigation pane of the EMR Serverless Spark page, click Artifacts.

4. On the Artifacts page, click Upload File.

5. Upload the sparkDataFrame-1.0-SNAPSHOT.jar package.

Step 5: Create and run a job