MaxCompute:Network connection process

Disclaimer

Network connection schemes

Prerequisites

Supported regions

Access over the Internet

Manage a public IP address or domain name on the Projects page

Access application

SQL statements that call UDFs

• Perform the following configurations:

  -- Set the odps.internet.access.list property to the public IP address or domain name and port number that are specified in the network connection application form. Specify the public IP address or domain name and port number in the following SQL statement. 
  -- If you want to add multiple IP addresses or domain names and port numbers, separate them with commas (,).
  set odps.internet.access.list=<ip_address:port|realm_name:port>;
  -- Execute the following SQL statement to call a UDF: 
  select <UDF_name>("<http://ip_address|realm_name>");

• ip_address:port | realm_name:port: required. The public IP address or domain name and port number that you want to access.

• UDF_name: the name of the UDF that you use to access the public IP address or domain name.

• The following sample code provides an example of the UDF code.

  package com.aliyun.odps.test.udf;
  import com.aliyun.odps.udf.UDF;
  import java.io.BufferedReader;
  import java.io.IOException;
  import java.io.InputStreamReader;
  import java.net.URL;
  public class <UDF_name> extends UDF {
      public String evaluate(String urlStr) throws IOException {
          URL url = new URL(urlStr);
          StringBuilder sb = new StringBuilder();
          try (BufferedReader reader = new BufferedReader(new InputStreamReader(url.openStream()))) {
              String line;
              while ((line = reader.readLine()) != null) {
                  sb.append(line).append('\n');
              }
          }
          return sb.toString();
      }
  }

• The UDF that is created based on the sample UDF code is named url_fetch. After the network connection application is approved, execute the following statements:

  set odps.internet.access.list=www.aliyun.com:80;
  select url_fetch("http://www.aliyun.com");

Spark on MaxCompute jobs

spark.hadoop.odps.cupid.smartnat.enable = true;
spark.hadoop.odps.cupid.internet.access.list=<ip_address:port>

IP address whitelist

If access control is configured for the service that you want to access, you must add the public outbound IP address of MaxCompute to the IP address whitelist of the service. To obtain the outbound IP address, submit a ticket.

Access over a VPC (dedicated connection)

1. Establish a dedicated network connection

   Procedure

   1. Perform authorization.

      • Grant the logon user the permission to establish network connections. The logon user must be the project owner or the user that is assigned the Super_Administrator role or Admin role at the tenant level. For more information about network connections, see Network connection. For more information about the roles, see Role planning. For more information about authorization, see Permissions on objects in a tenant.

      • Authorize MaxCompute to create ENIs in the VPC. This way, MaxCompute can be connected to the VPC. To grant the permissions, you need to use an Alibaba Cloud account to log on to the Alibaba Cloud Management Console, visit the Cloud Resource Access Authorization page, and click Confirm Authorization Policy.

        Note

        For more information about the billing rules when you access a VPC from MaxCompute, see Billing.

   2. Configure a security group. Create a security group for MaxCompute to manage the access of MaxCompute to various resources in the VPC.

      1. Log on to the VPC console. On the VPC page, click the ID of the destination VPC. On the page that appears, click the Resource Management tab.

      2. In the VPC Resources section of the Resource Management tab, move the pointer over the value of Security Group and click Add. On the Create Security Group page, click Create Security Group to create a security group for MaxCompute and record the ID of the security group. You must create a basic security group, instead of an advanced security group. By default, basic security groups allow outbound traffic, and advanced security groups do not allow outbound traffic. If you use an advanced security group, no objects in the VPC can be accessed. You must select the same VPC as the object that MaxCompute needs to access. For more information about how to create a security group, see Create a security group.

         • Create a security group

         • Configure a security group

      Note

      By default, MaxCompute automatically creates two ENIs based on the bandwidth requirements. You can use the two ENIs free of charge. The ENIs that are created by MaxCompute belong to the security group that you created. If you want to establish a connection between MaxCompute and an ApsaraDB for HBase cluster but access to the ApsaraDB for HBase cluster is not allowed in the security group, you can add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster. The IP addresses of the ENIs may change. We recommend that you add the CIDR block of the vSwitch to the whitelist to which the VPC belongs. To obtain the IP addresses of the ENIs, perform the following operations: Log on to the Elastic Compute Service (ECS) console. In the left-side navigation pane, click Elastic Network Interfaces in the Network & Security section to view the IP addresses of the ENIs.

   3. Establish a network connection between MaxCompute and the destination VPC.

      An Alibaba Cloud account or a RAM user that is assigned the tenant-level Super_Administrator or Admin role can establish a connection between MaxCompute and a VPC in the MaxCompute console. For more information about the roles, see Role planning. To establish a connection between MaxCompute and a VPC, perform the following steps:

      1. Log on to the MaxCompute console.

      2. In the left-side navigation pane, choose Tenants > Network Connection > Add Network Connection.

      3. In the Add Network Connection dialog box, configure the parameters and click OK. The following table describes the parameters.

         Parameter	Description
         Network Connection Name	The name of the custom network connection. The name must meet the following format requirements: Start with a letter. Contain only letters, underscores (_), and digits. Contain 1 to 63 characters in length.
         Type	The network connection type. Default value: Passthrough. Note The default value indicates that the VPC connection scheme is used.
         Region	The region in which you can use the VPC connection scheme to establish a network connection between MaxCompute and the specified object. For more information, see Supported regions.
         Selected VPC	The ID of the VPC. To obtain the ID of the VPC, perform the following operations: If you want to establish a network connection between MaxCompute and an ApsaraDB for HBase cluster or a Hadoop cluster, you can obtain the VPC ID in the network connection information in the console of the related service. In other cases, you can perform the following operations: Log on to the VPC console. On the VPC page, view the ID of the desired VPC in the Instance ID/Name column.
         vSwitch ID	The ID of the vSwitch to which the VPC belongs. To obtain the ID of the vSwitch, perform the following operations: If you want to establish a network connection between MaxCompute and an ApsaraDB for HBase cluster or a Hadoop cluster, you can obtain the vSwitch ID in the network connection information in the console of the related service. In other cases, you can perform the following operations: Log on to the VPC console. In the left-side navigation pane, click vSwitch. On the vSwitch page, click the name of the desired vSwitch. On the page that appears, view the vSwitch ID in the vSwitch Basic Information section.
         Security Group	The ID of the security group that is recorded in the Establish a dedicated network connection step.

   4. Configure the security group of the service that MaxCompute needs to access.

      After the ENI is enabled, you must add rules to the security group of the destination service to allow the MaxCompute security group created in Step 2 to access the destination service by using specific ports, such as port 9200 and port 31000. For example, if you want to access an ApsaraDB RDS instance, you need to add rules to the security group of the ApsaraDB RDS instance to allow access from the security group created in Step 2. If the service that you want to access does not support security groups, and only IP addresses can be added, you need to add the CIDR block of the vSwitch that is used by the destination service.

      • Configure the security group of the Hadoop cluster that MaxCompute needs to access.

        • To ensure that MaxCompute can access a Hadoop cluster, perform the following configurations for the security group of the Hadoop cluster:

          • Add inbound rules to the security group of the Hadoop cluster.

          • Set the authorization object to the security group to which the ENIs belong. In this case, the security group refers to the group that you created in Step 2.

          • Set the port number for the Hive metastore service to 9083.

          • Set the port number for NameNode of Hadoop Distributed File System (HDFS) to 8020.

          • Set the port number for DataNode of HDFS to 50010.

        • For example, if you want to allow MaxCompute to access a Hadoop cluster that is deployed on Alibaba Cloud E-MapReduce (EMR), you must configure the security group rules that are shown in the following figure. For more information, see Create a security group.

      • Configure the security group of an ApsaraDB for HBase cluster.

        • Add the security group that is created for MaxCompute to the security group of the ApsaraDB for HBase cluster or add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster.

        • For example, if you want to allow MaxCompute to access an ApsaraDB for HBase cluster, you can perform the following operations: Log on to the ApsaraDB for HBase console. On the Clusters page, click the name of the ApsaraDB for HBase cluster that you want to access in the ID / Name column. In the left-side navigation pane, click Access Control. Then, add the security group of MaxCompute on the Security Group tab or add the IP addresses of the ENIs created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster on the Whitelist Setting tab. For more information about how to add a security group or add an IP address to a whitelist, see Configure a whitelist and a security group.

          Note

          If the security group of MaxCompute cannot be added, you can add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB for HBase cluster on the Whitelist Setting tab. If the MaxCompute configuration is changed, the IP addresses of the ENIs may change. We recommend that you add the CIDR block of the vSwitch to the whitelist to which the VPC belongs.

      • Configure the security group of an ApsaraDB RDS instance.

        • Add the security group that is created for MaxCompute to the security group of the ApsaraDB RDS instance or add the IP addresses of the ENIs that are created by MaxCompute to a whitelist of the ApsaraDB RDS instance.

        • For example, if you want to allow MaxCompute to access an ApsaraDB RDS instance, you can perform the following operations: Log on to the ApsaraDB RDS console. On the Instances page, click the name of the ApsaraDB RDS instance that you want to access in the Instance ID/Name column. In the left-side navigation pane, click Whitelist and SecGroup. Then, add a security group on the Security Group tab or configure an IP address whitelist on the Whitelist Settings tab. For more information about how to add a security group or configure an IP address whitelist, see Configure a security group for an ApsaraDB RDS for MySQL instance or Configure an IP address whitelist.

        Note

        If the MaxCompute configuration is changed, the IP addresses of the ENIs may change. We recommend that you add the CIDR block of the vSwitch to the whitelist to which the VPC belongs.

2. SQL jobs

   Add configuration items.

   • For more information about the configurations for MaxCompute to access resources in a VPC by using UDFs, see Use UDFs to access resources in VPCs. The following code shows an example:

     -- Configure the name of the network connection that you established based on the VPC connection scheme. This setting is valid only for the current session.
     set odps.session.networklink=testLink;

   • Access resources in a VPC by using external tables. The following code shows an example:

     -- Configure parameters in the table creation statement.
      TBLPROPERTIES(
     'networklink'='<networklink_name>')

   • For more information about how to establish a network connection by using the data lakehouse solution, see Lakehouse of MaxCompute.

3. Spark on MaxCompute jobs

   • After you establish a dedicated network connection between MaxCompute and a VPC, add the following configurations to the SQL statement:

   • Add the following configurations to the SQL statement before you run a Spark job. The configurations allow MaxCompute to access objects in the destination VPC by using an ENI. For more information, see Access instances in a VPC from Spark on MaxCompute.

     • spark.hadoop.odps.cupid.eni.enable = true

     • spark.hadoop.odps.cupid.eni.info=regionid:vpc id

4. IP address whitelist

   If access control is configured for the object that you want to access, you must add the security group for the established dedicated network connection to the IP address whitelist of the object.

Access to specific Alibaba Cloud services

You can use this scheme if you want to access Alibaba Cloud services such as OSS, DLF, Tablestore, and Hologres by using SQL statements, UDFs, Spark, MapReduce, PyODPS, Mars, external tables, or the data lakehouse solution in MaxCompute. In this scheme, the endpoints of the cloud product interconnection network of Alibaba Cloud services are used.

Access OSS or Tablestore by using external tables

Access OSS or Tablestore by calling UDFs