DataWorks Deploy Center is an upgraded version of the existing task deployment feature. Deploy Center helps resolve deployment issues of objects such as nodes, functions, resources, and script templates in different environments. In most cases, if you want to create the same object at the source and destination, you can use Deploy Center to deploy the object from the source to the destination. This topic describes the use scenarios, deployment logic, and deployment processes of Deploy Center.
Scenarios
DataWorks Deploy Center provides cross-workspace deployment and cross-cloud deployment features based on different deployment environments.
Cross-workspace deployment
The cross-workspace deployment feature is formerly known as the cross-workspace clone feature. This feature is mainly used to deploy objects such as nodes, functions, resources, and script templates across workspaces under the same Alibaba Cloud account in the same region.
A workspace in basic mode provides only one environment. The development and production environments are not isolated. To isolate the development and production environments, you need to perform the following operations: Prepare two workspaces in basic mode. Workspace 1 serves as the development environment for testing, and Workspace 2 serves as the production environment for scheduling. Use the cross-workspace deployment feature to deploy tasks in Workspace 1 to Workspace 2. Common use scenarios of cross-workspace deployment:
Deploy an object from a workspace in basic mode to another workspace in basic mode.
Deploy an object from a workspace in basic mode to a workspace in standard mode.
Cross-cloud deployment
The cross-cloud deployment feature allows you to deploy objects such as nodes, functions, resources, and script templates across accounts, regions, or cloud platforms. The cloud platforms include Alibaba Finance Cloud and Alibaba Gov Cloud. In essence, the feature allows you to deploy objects from a source workspace to a destination workspace. The two workspaces belong to different regions, accounts, or cloud platforms.
Deployment logic
When you deploy a node in Deploy Center, the workflow to which the node belongs and the scheduling dependencies of the node are synchronized to the destination workspace. If scheduling dependencies are configured for a node, you can deploy the node only after ancestor nodes of the node are successfully deployed to the destination workspace. The following changes are made when you deploy a node:
The system modifies the input and output names that are prefixed with the source workspace name of the node. The source workspace name is replaced with the destination workspace name. If cross-workspace dependencies are configured for the node, you can configure dependency mappings for the deployment environment based on your business requirements. The dependencies and input and output names of the node after the deployment vary based on the configured dependency mappings. For more information, see Changes to dependencies and input and output names of nodes in this topic.
When you deploy tasks of the MaxCompute compute engine type, the system replaces the source workspace name in task code with the destination workspace name. For more information, see Changes to code of tasks of the MaxCompute compute engine type in this topic.
For information about how to configure dependency mappings, see Create and manage deployment environments. For information about how to configure scheduling dependencies, see Scheduling dependency configuration guide.
In this topic, output names are specified in the
Workspace name.Node nameformat for reference only. Actual output names may vary.
Changes to dependencies and input and output names of nodes
This section describes the changes that are made after nodes are deployed in different configuration scenarios.
No cross-workspace dependencies are configured.
No cross-workspace dependencies are configured for nodes in
project1. Deploy all nodes inproject1toproject2. After the deployment is complete, the project1 prefix in the input and output names of all nodes is replaced with project2.The input name of task_A is changed from
project1_roottoproject2_root.The output name of task_A is changed from
project1.task_A_outtoproject2.task_A_out.
Cross-workspace dependencies are configured, but no mappings are configured for the cross-workspace dependencies.
project1.task_Adepends onproject3.task_A. Deploy all nodes inproject1toproject2. Changes after node deployment:Node input and output names: The
project1prefix in the input and output names of all the nodes is changed toproject2.Cross-workspace dependencies:
project2.task_Adepends onproject3.task_A.
Cross-workspace dependencies are configured, and mappings are configured for the cross-workspace dependencies.
project1.task_Adepends onproject3.task_A. Deploy all nodes inproject1toproject2, and configure dependency mappings betweenproject3andproject4. Changes after node deployment:Node input and output names: The
project1prefix in the input and output names of all the nodes is changed toproject2.Cross-workspace dependencies:
project2.task_Adepends onproject4.task_A.
Changes to code of tasks of the MaxCompute compute engine type
After you deploy tasks of the MaxCompute compute engine type, such as ODPS SQL or ODPS Spark, the system replaces the source workspace name in task code with the destination workspace name when you run task code.
In this example, an ODPS SQL node named task_A is used. The code for querying table_A in project1 is Select * from project1.tableA. Deploy all nodes in project1 to project2. After nodes are deployed to project2, the code for querying table_A is changed to Select * from project2.tableA.
Limits
Limits on cross-workspace deployment:
Environments
DataWorks allows you to use the cross-workspace deployment feature only in workspaces in basic mode. You can deploy objects in a workspace in basic mode to another workspace in basic mode or a workspace in standard mode.
Permissions
Create a deployment environment: Only members to which the Workspace Administrator role is assigned can create and configure a deployment environment.
Create a deployment package: Only members to which the Development role is assigned can create a deployment package.
Deploy a deployment package: Only members to which the O&M, Deploy, or Workspace Administrator role is assigned can deploy a deployment package.
For information about how to add members to a workspace and grant permissions to the members, see Manage permissions on workspace-level services.
Others
The cross-workspace deployment feature allows you to deploy objects between different workspaces under the same Alibaba Cloud account in the same region. You cannot use this feature to deploy objects across regions. If you want to deploy objects across regions, use the cross-cloud deployment feature.
DataWorks does not allow you to deploy objects that belong to a workflow of an earlier version to a destination workspace. To deploy objects that belong to a workflow of an earlier version to a destination workspace, you must migrate the objects to the Business Flow section first.
Limits on cross-cloud deployment: The account that you want to use to perform the cross-cloud deployment operation must be a member of the source and destination workspaces, and is assigned the Workspace Owner or Workspace Administrator role in the source workspace. For information about how to add members to a workspace and grant permissions to the members, see Manage permissions on workspace-level services.
Deployment processes
Cross-workspace deployment
The following figure shows the cross-workspace deployment process. All operations are performed in the source workspace.
Create and configure a deployment environment. For more information, see Create and manage deployment environments.
The destination workspace to which you want to deploy nodes is a new deployment environment. Therefore, you must map information such as compute engine instances, resource groups, and dependencies between the source and the destination. After mappings are configured, you do not need to configure additional settings in the destination workspace when you deploy a deployment package, and the system replaces the environment information used by the nodes that you want to deploy with mapped environment information.
Add objects that you want to deploy to a deployment package and deploy the deployment package based on the created deployment environment. For information about how to create and deploy a deployment package, see Create a deployment package to deploy objects in the deployment package across workspaces.
NoteIn cross-workspace deployment scenarios, the system automatically deploys objects in a deployment package after you create the deployment package. You do not need to manually deploy the deployment package.
View the deployment result. For more information, see View deployment packages.
Cross-cloud deployment
The following figure shows the cross-cloud deployment process.
Create and configure a deployment environment. For more information, see Create and manage deployment environments.
The destination workspace to which you want to deploy nodes is a new deployment environment. Therefore, you must map information such as compute engine instances, resource groups, and dependencies between the source and the destination. After mappings are configured, you do not need to configure additional settings in the destination workspace when you deploy a deployment package, and the system replaces the environment information used by the nodes that you want to deploy with mapped environment information.
In the source workspace, add the objects that you want to deploy to a deployment package based on the created deployment environment and download the deployment package to your on-premises machine. Then, import the downloaded deployment package to the destination workspace, and commit and deploy the deployment package. For information about how to create and deploy a deployment package, see Create a deployment package to deploy objects in the deployment package across clouds.
NoteIn cross-cloud deployment scenarios, you must manually download the deployment package that you created to your on-premises machine and then import it to the destination workspace for deployment.
View the deployment result in the destination workspace. For more information, see View deployment packages.