DataWorks:Create and use a serverless resource group

Prerequisites

• You are familiar with the details about serverless resource groups, such as the specifications, performance, and billing standards. You have determined the specifications and subscription duration that you require based on your business scenario. For more information, see Overview of DataWorks resource groups and Billing of serverless resource groups.

• Serverless resource groups are supported in the following regions: China (Beijing), China (Shanghai), China (Shenzhen), China (Hangzhou), China (Hong Kong), China (Zhangjiakou), Singapore, Malaysia (Kuala Lumpur), Indonesia (Jakarta), Japan (Tokyo), UK (London), US (Silicon Valley), Germany (Frankfurt), and US (Virginia).

• You are granted the required permissions.

  • Only RAM users to which the AliyunBSSOrderAccess and AliyunDataWorksFullAccess policies are attached can purchase resource groups. For information about how to grant permissions to a RAM user and view the permissions of a RAM user, see Grant permissions to a RAM user and View the permissions of a RAM user.

  • Only a workspace administrator can associate a resource group with a workspace and change the workspace with which a resource group is associated.

  • For information about the permissions required to perform other operations on a resource group, see Custom policies used to manage permissions on the entities in the DataWorks console.

• If you want to use a serverless resource group in a virtual network operator (VNO) environment, you must first check whether your service provider allows you to purchase serverless resource groups.

Comparison between serverless resource groups and old-version resource groups

Precautions

• To ensure that your resource group can access the desired data source, such as a database, a data service, or other data in a specific network environment, you must establish a network connection between the resource group and the data source in advance based on the situations of the data source. For more information, see Network connectivity solutions.

  Important

  You can associate a serverless resource group with a virtual private cloud (VPC) to enable the resource group to access a data source or an address in a complex network environment over an internal network. By default, serverless resource groups cannot access the Internet. If you want to use a serverless resource group to access a data source or a network environment over the Internet, you must configure an Internet NAT gateway for the VPC with which the resource group is associated and associate the Internet NAT gateway with an elastic IP address (EIP). For more information, see the Scenario 5: Establish a network connection between a resource group and a data source that is deployed on the Internet section in this topic.

• If you have associated a serverless resource group with a VPC and a vSwitch, do not modify the configurations of the VPC and vSwitch. Otherwise, DataWorks tasks that run on the serverless resource group may fail.

Billing of serverless resource groups

For information about billing of serverless resource groups, see Billing of serverless resource groups.

Step 1: Create a serverless resource group

1. Go to the Resource Groups page.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, click Resource Group.

2. On the Exclusive Resource Groups tab, click Create Resource Group to go to the buy page of serverless resource groups.

   Parameter	Description
   Region and Zone	The region in which you want to create the resource group. The region must be the same as the region in which the workspace resides.
   Billing Method	Subscription: You must pay for the resource group before you use it. Pay-as-you-go: You can use the resource group before you pay for it.
   Resource Group Specifications	This parameter is required only if you set the Billing Method parameter to Subscription. Valid values: 2 to 99999999. Unit: CU. Note 1 CU = 1 vCPU core + 4 GiB of memory. For information about purchase suggestions and the minimum specifications that are required to run different types of tasks, see Performance metrics and purchase suggestions. The value 99999999 indicates that the number of CUs you can purchase is not limited. However, the number of CUs that you can purchase may be affected by the inventory. If the inventory is insufficient, you must pay attention to the prompt that is displayed when you purchase a serverless resource group.
   Resource Group Name	The name of the resource group.
   Resource Group Description	The description of the resource group.
   VPC	The VPC and vSwitch with which you want to associate the resource group. You can select a VPC based on the network that the resource group needs to access. If the resource group needs to access a data source that belongs to the same Alibaba Cloud account and resides in the same region as the resource group, you can select the VPC where the data source resides and a vSwitch in the VPC. If the resource group needs to access a data source that resides in a complex network environment, you must use a connection tool such as VPN Gateway or Express Connect to establish a network connection between the VPC with which the resource group is associated and the VPC where the data source resides. For more information, see Network connectivity solutions. Note If no VPCs or vSwitches are available, you must go to the VPC console to create a VPC or a vSwitch. For more information about VPCs, see What is a VPC? You can associate the resource group with one or more other VPCs after the resource group is created. If you set the Billing Method parameter to Subscription and the VPC that you select is used in DataService Studio, data computing, and data synchronization, you cannot associate the resource group with another VPC or change the associated VPC for the resource group when you use the resource group in DataService Studio. Make appropriate planning in advance. If you have associated a serverless resource group with a VPC and a vSwitch, do not modify the configurations of the VPC and vSwitch. Otherwise, DataWorks tasks that run on the serverless resource group may fail.
   vSwitch
   Billing Cycle	The subscription duration of the resource group. This parameter is required only if you set the Billing Method parameter to Subscription. Important To prevent your business from being affected due to service suspension or resource release when the resource group expires, we recommend that you select Auto-renewal. After you select Auto-renewal, fees are automatically deducted from your Alibaba Cloud account based on the actual prices before the resource group expires. The auto-renewal cycle is one month.
   Service-linked Role	The service-linked role. The first time you create a serverless resource group, you must create a service-linked role named AliyunServiceRoleForDataWorks. When you create a serverless resource group in subsequent operations, the system automatically assigns the service-linked role. Note The service-linked role AliyunServiceRoleForDataWorks is used to access resources in a VPC, in an elastic network interface (ENI), and in a security group. For more information about the service-linked role, see DataWorks service-linked role.

Step 2: Associate the resource group with a workspace

After the resource group is created, you must associate the resource group with a workspace. Then, you can select the resource group when you create tasks in the workspace.

• Associate the resource group when you create a workspace

  1. Log on to the DataWorks console. In the top navigation bar, select the desired region. Then, click Workspace in the left-side navigation pane.

  2. On the Workspaces page, click Create Workspace. On the Create Workspace page, select the created resource group from the Default Resource Group drop-down list.

• Associate the resource group with an existing workspace

  1. Go to the Resource Groups page.

     Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, click Resource Group.

  2. On the Resource Groups page, find the created resource group and click Associate Workspace in the Actions column. In the Associate Workspace panel, find the workspace with which you want to associate the resource group and click Associate in the Actions column.

Step 3: Configure network connectivity

To ensure that your task can run as expected, you must complete network connectivity configuration. This way, the resource group can access the desired data source. For more information, see Network connectivity solutions.

You can configure an internal DNS resolution record for the VPC with which the serverless resource group is associated. This way, the serverless resource group can access the related data source by using a custom internal domain name. For example, if you associate the serverless resource group with the VPC in which a CDH cluster is deployed and configure an internal DNS resolution record for the VPC, the serverless resource group can access the CDH cluster by using a custom domain name. For more information, see Preparations: Obtain configuration information about a CDH or CDP cluster and configure network connectivity.

Step 4: Modify configuration items for the resource group

Next step: Configure the resource group for different tasks

After the resource group is created and configured, you need to configure the resource group for data synchronization, data scheduling, and DataService Studio tasks to use the resource group to run the tasks. For more information, see General reference: Change the resource groups used by tasks.

Other operations

References

• For more information about resource groups, see Overview of DataWorks resource groups.

• You can use the intelligent monitoring feature provided in Operation Center to monitor the resource usage of a resource group and the number of instances that are waiting for resources in a resource group. For more information about how to use the intelligent monitoring feature, see Create a custom alert rule.

• When you view the status of a resource group on the Resource Groups page, take note of the following items:

  • If the resource group is expired, you can click the icon in the Actions column and then click Renew to renew the resource group.

  • If the resource usage of the resource group reaches the warning threshold, you can click the icon in the Actions column and then click Scale Out to scale out the resource group. For more information, see the Scale out or in a resource group section in this topic.

• If a specific development environment, such as an environment with third-party library dependencies, is required for running your tasks on a serverless resource group, you can create a custom image that contains all required development packages and dependencies. Then, you can use the custom image as the runtime environment when you run tasks on the serverless resource group.