AnalyticDB:Configure Spark application to access the public Internet

Background information

Internet NAT gateways provide the public IP address translation, enabling instances in a virtual private cloud (VPC) to access the Internet and provide Internet-facing services. For more information, see What is an Internet NAT gateway?

Precautions

If a self-managed database or third-party cloud service providers have network security controls such as firewalls and IP whitelists, you must manually add either the public IP addresses of the SNAT entry or the CIDR block of the vSwitch to these security settings to enable successful access from your Spark application.

Procedure

1. Create an Internet NAT gateway.

   The Internet NAT gateway must reside in the same region as the AnalyticDB for MySQL cluster.

2. Associate an EIP with the Internet NAT gateway.

3. Create an SNAT entry.

   We recommend selecting Specify vSwitch for the SNAT entry.

4. If you are developing Spark jobs using a Notebook, you can skip this step. Otherwise, configure the following key parameters in your Spark application, as shown in the example below.

   {
       "comments": ["-- Here is just an example of SparkPi. Modify the content and run your spark program."],
       "args": ["1000"],
       "file": "local:///tmp/spark-examples.jar",
       "name": "SparkPi",
       "className": "org.apache.spark.examples.SparkPi",
       "conf": {
           "spark.driver.resourceSpec": "small",
           "spark.executor.instances": 1,
           "spark.executor.resourceSpec": "small",
           "spark.adb.eni.enabled": "true",
           "spark.adb.eni.vswitchId": "vsw-bp1ghmwrkeaw3xvnd****",
           "spark.adb.eni.securityGroupId": "sg-bp1airvjxl5vpr2****"	
       }
   }

   Parameters:

   Key parameter	Description
   spark.adb.eni.enabled	Specifies whether to enable Elastic Network Interface (ENI) when accessing external data sources via external tables. Set this parameter to true.
   spark.adb.eni.vswitchId	The vSwitch ID of the ENI. Set this parameter to the vSwitch ID that is specified when you create an SNAT entry.
   spark.adb.eni.securityGroupId	The security group ID of the ENI. Set this parameter to the ID of the security group in the same VPC as the NAT gateway.

   For more information about Spark applications, see Overview of Spark application development.