DataWorks provides exclusive resource groups for Data Integration, allowing you to allocate dedicated computing resources to your synchronization tasks. This enhances the efficiency and stability of task execution. Before utilizing your purchased exclusive resource group for Data Integration, you may need to configure network settings and IP address whitelists. This topic describes the steps from purchasing to using the resource group.
If you have never activated DataWorks before June 10, 2024, you can purchase and use only serverless resource groups after you activate DataWorks, and you cannot purchase or use old-version resource groups.
Prerequisites
You need to understand the specifications and performance of exclusive resource groups for Data Integration (how many tasks can run simultaneously with different specifications), billing types (how different specifications are charged), and other details. Plan the specifications and subscription duration of the resource group you need to purchase based on your business scenario in advance. For more information, see exclusive resource group for Data Integration billing description: subscription.
Notes
Exclusive resource groups for Data Integration support data synchronization in complex network environments, such as across Alibaba Finance Cloud, Alibaba Gov Cloud, and between different Alibaba Cloud accounts or data centers. To ensure the data synchronization task process requirements are met, establish network connections between your resource group and data sources, and configure IP address whitelists for data source accessibility. Use Data Integration only after confirming network connectivity. If the network is not connected, the data synchronization task cannot be executed. For more information about network connectivity solutions and whitelist considerations between exclusive resource groups for Data Integration and data sources, see exclusive resource group for Data Integration.
-
If you do not need to connect your resource group to your data source but want to only resolve task latency issues caused by insufficient resources of the shared resource group for Data Integration, you do not need to pay attention to the network settings described in this topic. You can purchase an exclusive resource group for Data Integration in any zone and do not need to configure network settings.
-
DataWorks exclusive resource groups for Data Integration can access the Internet by default. Because shared bandwidth is used, the quality of Internet access cannot be guaranteed. If you have a strong dependency on the Internet, use Serverless resource groups.
Limits
-
Only users with AliyunBSSOrderAccess and AliyunDataWorksFullAccess permissions can purchase resource groups.
-
Only workspace administrators can associate a resource group with a workspace and change the workspace with which a resource group is associated.
-
For more information about the permissions of the resource group interface, see resource group permission control policy.
-
For more information about creating custom policy content and authorization operations, see create a custom policy (optional).
-
An exclusive resource group for Data Integration with specifications of 4 vCPUs and 8 GiB of memory can be associated with a maximum of two VPCs. An exclusive resource group for Data Integration with other specifications can be associated with a maximum of three VPCs.
Step 1: purchase a resource group
DataWorks resource groups offer subscription services for exclusive resource groups, which must be purchased separately. You can purchase them based on the following information.
Only users with AliyunBSSOrderAccess and AliyunDataWorksFullAccess permissions can purchase resource groups.
-
Log on to the DataWorks console or the .
-
In the left-side navigation pane, click Resource Groups to access the Resource Group List page. On the Exclusive Resource Group tab, click Create Legacy Integration Resource Group. On the purchase page, set the parameters according to your business needs. Below are the main points for configuration.
Parameter
Description
Region And Zone
Select the region in which you want to use the exclusive resource group.
NoteAn exclusive resource group for Data Integration cannot be used across regions. For example, exclusive resource groups in the China (Shanghai) region can be used only by the workspaces in the China (Shanghai) region.
Exclusive Resource Type
Select Exclusive Data Integration Resource.
Resource Name
The name of the exclusive resource group. The name must be unique within a tenant. Otherwise, an error is reported when you click OK.
NoteA tenant refers to an Alibaba Cloud account. Each tenant can have multiple RAM users.
Resource Remarks
The description of the exclusive resource group.
Billing Cycle
Exclusive resource groups are provided as a subscription service. To ensure that the service is not interrupted, we recommend that you select Auto-renewal Upon Expiration. After the resource group is created, you can also go to the Alibaba Cloud Renewal Management page to enable or disable the auto-renewal service. For more information, see general reference: stop using DataWorks products.
You can configure other parameters based on your business requirements.
-
Click Buy Now. Then, follow the on-screen instructions to complete the payment and purchase the exclusive resource group for Data Integration.
After the purchase, DataWorks starts to initialize the resource group. When the resource group enters the Running state, the resource group is created in the DataWorks console.
NoteThe initialization of the exclusive resource group takes about 20 minutes. Please wait patiently for its status to update to Running.
After the exclusive resource group is created in the DataWorks console, you must associate the resource group with a workspace. This way, you can select the resource group when you configure a task in the workspace.
Step 2: associate with a workspace
An exclusive resource group for Data Integration must be associated with its workspace before it can be used in the corresponding workspace. An exclusive resource group for Data Integration can be assigned to multiple workspaces but cannot be used across regions. For example, an exclusive resource group for Data Integration in the Shanghai region can be associated only with workspaces in the Shanghai region and cannot be associated with workspaces in other regions. The following are the steps to associate the resource group with a workspace.
Only workspace administrators can associate a resource group with a workspace and change the workspace with which a resource group is associated.
-
Log on to the DataWorks console or the .
-
On the Resource Group List page, click the Exclusive Resource Group tab. Then, click Associate With Workspace next to the corresponding resource group.
-
On the Associate With Workspace page, you can click Associate next to the corresponding workspace to link the exclusive resource group to it.
Step 3: network configuration
Associate with a VPC
Exclusive resources are deployed in the virtual private cloud (VPC) in which DataWorks is hosted. Exclusive resources are disconnected from other network environments. To use exclusive resources, you must configure network settings to associate the exclusive resources with a VPC that can connect to data sources. This way, the exclusive resources can access the data sources over the VPC. The following are the steps to associate with a VPC.
An exclusive resource group for Data Integration with specifications of 4 vCPUs and 8 GiB of memory can be associated with a maximum of two VPCs. An exclusive resource group for Data Integration with other specifications can be associated with a maximum of three VPCs.
-
Log on to the DataWorks console or the .
-
On the Resource Group List page, click the Exclusive Resource Group tab. Then, click Network Settings next to the corresponding resource group to access the VPC association page.
Before associating the exclusive resource group with a VPC, log on to the RAM console with your Alibaba Cloud account and authorize DataWorks to access your cloud resources. You can authorize access on the cloud resource access authorization page. You can also authorize access in the pop-up dialog box that appears when you first enter the control panel with your Alibaba Cloud account.
-
Associate with a VPC.
-
Click Add Association in the upper-left corner of the VPC Association page. In the Add VPC Association dialog box, set the necessary parameters. Below are the configuration instructions for various network environments.
NoteIf you want to use the resource group to access a data source, such as an Alibaba Cloud data source or a self-managed data source hosted on an ECS instance, you can select a network connectivity solution and configure network settings based on whether the resource group and data source belong to the same Alibaba Cloud account.
Parameter
Configuration instructions (data source and exclusive resource in the same account and region)
Configuration instructions (data source and exclusive resource in different accounts or regions)
Associate With A VPC
If your data source and the exclusive resource group belong to the same Alibaba Cloud account, we recommend that you select the VPC in which your data source resides.
If they do not belong to the same Alibaba Cloud account, configure this parameter based on the description for the scenario where your data source and the exclusive resource group belong to different regions.
If your data source and the exclusive resource group belong to different regions or Alibaba Cloud accounts, you must select a VPC that connects to the data source. For example, if your data source is not in the Alibaba Cloud VPC network environment, you can click Create VPC to create a VPC for the exclusive resource group. After the VPC is created, set this parameter to the newly created VPC or select a VPC that connects to the target database network.
NoteIf the DataWorks workspace and the data source do not belong to the same region or Alibaba Cloud account, you must use a VPN Gateway or Express Connect to connect the VPC associated with the exclusive resource group to the VPC in which the data source resides. You must also manually add a route that points to the IP address of the target database to ensure connectivity between the two networks. For more information, see network connectivity solutions.
Zone
Select the zone in which your data source resides.
Select a zone from which a network connection to your data source is established.
Vswitch
If you set the VPC parameter to the VPC in which your data source resides, we recommend that you select the vSwitch to which the data source is connected.
NoteAfter you associate the exclusive resource group with the VPC in which the data source resides and a vSwitch that resides in the VPC, a route that points to the CIDR block of the VPC is automatically added. This ensures that the exclusive resource group can access the data sources in this VPC.
Select a vSwitch from which a network connection to your data source is established. If no vSwitch is available, you can click Create Vswitch to create a vSwitch for the exclusive resource group. After the vSwitch is created, set this parameter to the newly created vSwitch.
-
Click OK to complete the VPC association.
NoteIf your data source and the exclusive resource group belong to different regions or Alibaba Cloud accounts, after you associate the exclusive resource group with a VPC, you must add a route that points to the IP address of your data source.
-
-
Optional: Configure Host.
You may fail to access your data source by using IP addresses. For example, you can access your data source only by using hostnames. In this case, you must perform the following steps to add host configurations. Otherwise, the connectivity test fails when you add the data source by using its hostnames.
-
Click Host Configuration. At the top-left corner of the Host Configuration page, click Add. In the Add Domain Name Host Configuration dialog box, set the parameters as described below.
Parameter
Configuration description
IP Address
The actual IP address of the data source.
Host Domain Name
The hostname that is used to access the data source. If you want to specify multiple hostnames, place each hostname on a separate line.
-
To add multiple IP addresses, click Add to add them.
Note-
The IP address or hostnames that are added in a host configuration must be different from the IP addresses or hostnames in existing host configurations.
-
You can map one IP address to multiple hostnames in a host configuration. However, one hostname can point to only one IP address.
-
-
Add to whitelist
If the exclusive resource group for Data Integration and the data source reside in the same zone, but the network cannot be connected in the VPC and vSwitch, the data source may have whitelist restrictions. Add the following content to the whitelist of the data source.
-
If the exclusive resource group accesses your data source over an internal network, you must add the CIDR block of the vSwitch with which the resource group is associated to the IP address whitelist of your data source.
After you associate the exclusive resource group with a VPC, on the Resource Group List page, click the Exclusive Resource Group tab. Click Network Settings next to the corresponding resource group to go to the VPC Association tab and view the Vswitch CIDR Block.
-
If the exclusive resource group accesses your data source over the Internet, you must add the EIP Address of the exclusive resource group to the IP address whitelist of your data source.
Step 4: test network connectivity
After completing the network configuration, test the network connectivity between the resource group and your data source by performing the following operations.
Go to the Data Sources page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Management Center.
In the left-side navigation pane of the SettingCenter page, click Data Sources.
-
Click the Edit button in the Actions column next to the relevant data source.
-
On the Data Integration tab, click Test Connectivity next to the name of the corresponding resource group. If the connectivity status is Connected, the connection has been established successfully.
NoteIf the network is not connected, click Self-troubleshoot to use the diagnostic tool to diagnose network connectivity issues. For more information about the network connectivity between data sources and exclusive resource groups in different network environments, see network connectivity solutions.
-
Click Complete.
More operations
View resource group usage and resource group Service Monitoring
You can view the resource usage and the number of instances awaiting resources in a resource group through the DataWorks console. Additionally, the Operation Center's intelligent monitoring feature allows you to monitor these metrics for the resource group. For more information about viewing resource group usage, see view the usage of an exclusive resource group. For details on resource group monitoring, see create a custom rule.
Replace the zone of a resource group
To change the zone of a resource group, perform the following steps:
-
Log on to the DataWorks console or the .
-
In the left-side navigation pane, click Resource Groups. On the Resource Group List page, click the Exclusive Resource Group tab. Locate the resource group with the Purpose of Data Integration.
-
Click the Operation column next to the resource group, select the
icon, choose Zone Replacement, and enter the Replace Resource Group Zone dialog box. -
In the Replace Resource Group Zone dialog box, select the Current Zone and Number of Machines for the resource group you want to change. Then, choose the new Replace With Zone and Number of Machines to Replace.
-
Click Confirm Replacement to finalize the zone replacement for the resource group.
Changing the zone of a resource group may result in network alterations:
-
CIDR blocks of the resource group: Each zone of an ECS instance within a resource group has a unique CIDR block. The CIDR blocks of the resource group may change if the zone is altered.
-
Primary ENI IP addresses of the resource group: The primary ENI IP address of an ECS instance will change if its zone changes, with a new IP address from the CIDR block of the new zone being assigned.
-
Elastic IP address (EIP) associated with the resource group: If the EIP associated with the resource group is added to the IP address whitelist of a data source, you must update the whitelist after the zone change to ensure continued access and operation on the data source.
Appendix: change the resource group for Data Integration
After creating and configuring an exclusive resource group for Data Integration, you can switch the resource group used by nodes to a different one using the following methods.
|
Operating Environment |
Supported Change Operations |
Interface Entry |
|
Change the resource group for a node in the production environment |
Change the resource groups for multiple nodes simultaneously |
Go to the page to configure the settings.
Select the tasks whose resource groups you want to change. Click in the bottom menu bar to change the resource groups for multiple nodes simultaneously. |
|
Change the resource group for a node in the development environment |
|
Go to the DataStudio page to configure the settings.
Note
If you cannot find the entry point for changing the resource groups for nodes, select Offline synchronization from the Node Type drop-down list in the filter condition section to search for all batch synchronization nodes. |
icon. Select the tasks whose resource groups you want to change. Click Change Data Integration Resource Group in the bottom menu bar to change the resource groups for multiple nodes simultaneously. 