MaxCompute:Network connection process

SQL UDFs jobs

• Use the following settings:

  -- Set the public IP address or endpoint and port number that are configured in the network connection application form. Specify the public IP address or endpoint and port number in the following SQL statement.
  -- If you want to add multiple endpoints and port numbers, separate them with commas (,)
  SET odps.internet.access.list=<ip_address:port|realm_name:port>;
  -- Execute the following SQL statement to call a UDF.
  SELECT <UDF_name>("<http://ip_address|realm_name>");

• ip_address:port | realm_name:port: Required. Specify the public IP address or endpoint and port number you want to access.

• UDF_name: The UDF used to access the public IP address or endpoint.

• UDF code example:

  package com.aliyun.odps.test.udf;
  import com.aliyun.odps.udf.UDF;
  import java.io.BufferedReader;
  import java.io.IOException;
  import java.io.InputStreamReader;
  import java.net.URL;
  public class <UDF_name> extends UDF {
      public String evaluate(String urlStr) throws IOException {
          URL url = new URL(urlStr);
          StringBuilder sb = new StringBuilder();
          try (BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()))) {
              String line;
              while ((line = reader.readLine()) != null) {
                  sb.append(line).append('\n');
              }
          }
          return sb.toString();
      }
  }

• Execution: After the network connection is approved, execute the following sample command, where the UDF created from the sample code is named url_fetch:

  SET odps.internet.access.list=www.aliyun.com:80;
  SELECT url_fetch("http://www.aliyun.com");

Spark on MaxCompute jobs

Add the following configuration to the Spark client's conf file or the Spark job configuration submitted through DataWorks.

spark.hadoop.odps.cupid.smartnat.enable = true;
spark.hadoop.odps.cupid.internet.access.list=<ip_address:port>

Step 1: Authorization

• Authorize the user: Grant the permissions to create network connection objects to the logon user. For details, see Networklink. The authorized user must be the project owner or a user with the Super_Administrator or Admin role at the tenant level. For more information, see Role planning and Permissions on objects in a tenant.

• Authorize MaxCompute: This authorization allows MaxCompute to create ENIs in the VPC to establish a connection between MaxCompute and the VPC. To grant the permissions, you must use an Alibaba Cloud account to log on to the Alibaba Cloud Management Console, visit the Cloud Resource Access Authorization page, and click Confirm Authorization Policy.

  Note

  For more information about the billing rules when you access a VPC from MaxCompute, see VPC peering connections.

Step 2: Configure security group rules

Create a security group for MaxCompute to manage the access of MaxCompute to various resources in the VPC.

1. Log on to the VPC console. On the VPC page, click the ID of the destination VPC. On the page that appears, click the Resource Management tab.

2. In the VPC Resources section of the Resource Management tab, move the pointer over the value of Security Group and click Add. On the Create Security Group page, click Create Security Group to create a security group for MaxCompute and record the ID of the security group. You must create a basic security group, not an advanced one. By default, basic security groups allow outbound traffic, while advanced ones do not. If you use an advanced security group, no objects in the VPC can be accessed. You must select the same VPC as the object that MaxCompute needs to access. For more information about how to create a security group, see Create a security group.

   • Create a security group

   • Configure the security group. For security group configuration methods, see Security groups for different use cases.

Step 3: Create a network connection between MaxCompute and the destination VPC

An Alibaba Cloud account or a RAM user that is assigned the tenant-level Super_Administrator or Admin role can establish a connection between MaxCompute and a VPC in the MaxCompute console. For more information about the roles, see Role planning. To establish a connection between MaxCompute and a VPC, perform the following steps:

1. Log on to the MaxCompute console.

2. In the left-side navigation pane, choose Tenants > Network Connection > Add Network Connection.

3. In the Add Network Connection dialog box, configure the parameters and click OK. The following table describes the parameters:

   Parameter	Description
   Network Connection Name	The name of the custom network connection. The name must meet the following format requirements: Starts with a letter. Contains only letters, underscores (_), and digits. Is 1 to 63 characters in length.
   Type	The network connection type. Default value: Passthrough. Note The default value indicates that the VPC connection scheme is used.
   Region	The region in which you can use the VPC connection scheme to establish a network connection between MaxCompute and the specified object. For more information, see Supported regions.
   Selected VPC	The ID of the VPC. To obtain the ID of the VPC, perform the following operations: If you want to establish a network connection between MaxCompute and an ApsaraDB for HBase cluster or a Hadoop cluster, you can obtain the VPC ID in the network connection information in the console of the related service. In other cases, you can perform the following operations: Log on to the VPC console. On the VPC page, view the ID of the desired VPC in the Instance ID/Name column.
   vSwitch ID	The ID of the vSwitch to which the VPC belongs. To obtain the ID of the vSwitch, perform the following operations: If you want to establish a network connection between MaxCompute and an ApsaraDB for HBase cluster or a Hadoop cluster, you can obtain the vSwitch ID in the network connection information in the console of the related service. In other cases, you can perform the following operations: Log on to the VPC console. In the left-side navigation pane, click vSwitch. On the vSwitch page, click the name of the desired vSwitch. On the page that appears, view the vSwitch ID in the vSwitch Basic Information section.
   Security Group	The ID of the security group that is recorded in the Establish a dedicated network connection step.

Step 4: Configure the security group of the destination service

After the preceding operations are complete and the dedicated connection is established, you must add rules to the security group of the destination service to allow the MaxCompute security group created in Step 2 to access the destination service by using specific ports, such as port 9200 and port 31000.

For example, if you want to access an ApsaraDB RDS instance, you must add rules to the security group of the ApsaraDB RDS instance to allow access from the security group created in Step 2. If the service that you want to access does not support security groups, and only IP addresses can be added, you must add the CIDR block of the vSwitch that is used by the destination service.

• Configure the security group of the Hadoop cluster that MaxCompute needs to access.

  • To ensure that MaxCompute can access a Hadoop cluster, perform the following configurations for the security group of the Hadoop cluster:

    • Add inbound rules to the security group of the Hadoop cluster.

    • Set the authorization object to the security group to which the ENIs belong. In this case, the security group refers to the group that you created in Step 2.

    • Set the port number for the Hive metastore service to 9083.

    • Set the port number for NameNode of Hadoop Distributed File System (HDFS) to 8020.

    • Set the port number for DataNode of HDFS to 50010.

  • For example, if you want to allow MaxCompute to access a Hadoop cluster that is deployed on Alibaba Cloud E-MapReduce (EMR), you must configure the security group rules that are shown in the following figure. For more information, see Create a security group.

• Configure the security group of an ApsaraDB for HBase cluster.

  • Add the security group that is created for MaxCompute to the security group of the ApsaraDB for HBase cluster or add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster.

  • For example, if you want to allow MaxCompute to access an ApsaraDB for HBase cluster, you can perform the following operations: Log on to the ApsaraDB for HBase console. On the Clusters page, click the name of the ApsaraDB for HBase cluster that you want to access in the ID/Name column. In the left-side navigation pane, click Access Control. Then, add the security group of MaxCompute on the Security Group tab or add the IP addresses of the ENIs created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster on the Whitelist Setting tab. For more information about how to add a security group or add an IP address to a whitelist, see Configure a whitelist and a security group.

    Note

    If the security group of MaxCompute cannot be added, you can add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster on the Whitelist Setting tab. If the MaxCompute configuration is changed, the IP addresses of the ENIs may also change. We recommend that you add the CIDR block of the vSwitch to the whitelist to which the VPC belongs.

• Configure the security group of an ApsaraDB RDS instance.

  • Add the security group that is created for MaxCompute to the security group of the ApsaraDB RDS instance or add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB RDS instance.

  • For example, if you want to allow MaxCompute to access an ApsaraDB RDS instance, you can perform the following operations: Log on to the ApsaraDB RDS console. On the Instances page, click the name of the ApsaraDB RDS instance that you want to access in the Instance ID/Name column. In the left-side navigation pane, click Whitelist and SecGroup. Then, add a security group on the Security Group tab or configure an IP address whitelist on the Whitelist Settings tab. For more information about how to add a security group or configure an IP address whitelist, see Configure a security group for an ApsaraDB RDS for MySQL instance or Configure an IP address whitelist.

  Note

  If the MaxCompute configuration is changed, the IP addresses of the ENIs may also change. We recommend that you add the CIDR block of the vSwitch to the whitelist to which the VPC belongs.

Access the VPC by using SQL statements

• Access resources in a VPC by using UDFs. For details, see Use UDFs to access resources in VPCs. The following code shows an example:

  -- Configure the name of the network connection that you established based on the VPC connection scheme. This setting is valid only for the current session.
  SET odps.session.networklink=testLink;

• Access resources in a VPC by using external tables. The following code shows an example:

  -- Configure parameters in the table creation statement.
   TBLPROPERTIES(
  'networklink'='<networklink_name>')

• Configure Networklink for the data lakehouse solution. For details, see Lakehouse of MaxCompute.

Access the VPC by using Spark

To run a Spark job, add the following configurations to use the ENI dedicated connection for accessing services in the destination VPC. For details, see Access instances in a VPC from Spark on MaxCompute.

• spark.hadoop.odps.cupid.eni.enable = true

• spark.hadoop.odps.cupid.eni.info=regionid:vpc id