Data Governance Center provides a comprehensive undeployment solution for invalid tasks or tables that are identified during data governance. The solution supports various features, such as undeployment risk evaluation, undeployment notification push, and undeployment process management. This solution eliminates the need to manually identify undeployment risks. This solution also resolves the issue that you cannot undeploy tasks whose descendant tasks are scheduled as normal or you cannot determine whether to undeploy tasks due to unpredictable risks. The solution allows you to undeploy multiple tasks or tables at the same time. This improves the undeployment efficiency.
Background information
Graceful undeployment supports undeployment impact analysis and undeployment process management, which are unavailable in the traditional undeployment. Task undeployment notifications can be sent to related contacts by using various methods at each phase of the task undeployment process.
Operation | Undeployment method | Comparison | Description |
Direct undeployment | To directly undeploy a task, go to Operation Center. In the left-side navigation pane, choose Auto Triggered Node O&M > Auto Triggered Nodes. On the page that appears, find the task that you want to manage, click More, and then click Freeze in the Actions column. |
| |
Graceful undeployment | You can manually specify the objects that you want to undeploy or select the objects that are automatically recommended by the system. You can evaluate the impact scope and risks of the undeployment. | ||
Direct undeployment | None |
| |
Graceful undeployment | You can send notifications to the specified contacts in each phase of the graceful undeployment process by internal messages, emails, DingTalk chatbots, or webhooks. | ||
Direct undeployment | None |
| |
Graceful undeployment |
| ||
Direct undeployment | None |
| |
Graceful undeployment | You can view the details of an undeployment plan and perform various operations on the tasks that are specified in the plan. For example, you can view the details of the tasks, cancel the task undeployment, or roll back the undeployment operation. |
Limits
Only workspace administrators, tenant-level or workspace-level data governance administrators, and O&M personnel can create undeployment plans. For more information about how to assign roles to workspace members, see the "Add a RAM user to a workspace as a member and assign roles to the member" section in the Manage permissions on workspace-level services topic.
You can add only objects in the same workspace to an undeployment plan.
Only Alibaba Cloud accounts, RAM users to which the AliyunDataWorksFullAccess policy is attached, and workspace administrators can perform all operations on an undeployment plan. O&M personnel have full permissions only on the undeployment plans that they create. O&M personnel have only read permissions on the undeployment plans that are created by other users. Tenant-level or workspace-level data governance administrators can create, configure, and delete undeployment plans but cannot start or execute tasks on which they do not have permissions.
Up to 200 objects can be added to an undeployment plan.
The graceful undeployment feature allows you to roll back the operations performed on a task before the Undeploy Task phase of the process, and allows you to roll back the operations performed on a table before the Undeploy Completed phase of the process.
Usage notes
You cannot select an object that is already added to an undeployment plan. Each object can be added only to one undeployment plan.
Specific types of objects cannot be added to an undeployment plan. For example, you cannot add the root node of a workspace to an undeployment plan or add a zero load node that you created for an undeployment plan to the undeployment plan.
If an object that you want to undeploy has descendant objects and the scheduling dependencies between the descendant objects and the object are not removed within the specified period of time, the system generates a zero load node and configures the zero load node as the ancestor node of the descendant objects.
The operations that you need to perform to create a node undeployment plan vary based on the type of node. You can follow the on-screen instructions to create a node undeployment plan for a type of node.
The graceful undeployment feature allows you to roll back an undeployment operation only before the object is undeployed.
Graceful undeployment procedure
Step 1: Create an undeployment plan
Go to the Data Governance Center page.
Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose . On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.
In the upper-left corner of the DataStudio page, click the
icon and choose .
In the top navigation bar of the Data Governance Center page, click Workbench. The Governance issues page appears.
In the left-side navigation pane, click Graceful Shutdown.
In the upper part of the Graceful Shutdown page, select the workspace to which the objects that you want to undeploy belong from the Workspace drop-down list. Only the workspaces to which you are added as a member are displayed in the drop-down list.
To join a workspace, you must contact the administrator of the workspace. For more information, see the "Add a RAM user to a workspace as a member and assign roles to the member" section in Manage permissions on workspace-level services.
On the Graceful Shutdown page, click Create Plan.
You can also find the nodes that you want to undeploy from the workspace perspective on the Governance issue page, and click Create Plan in the Actions column of the nodes to go to the Create Plan page.
Step 2: Select the objects that you want to undeploy
Select the objects that you want to undeploy
In the To-Be-Undeployed Object section, you can select deployed tasks or tables based on your business requirements.
Undeploy tasks: Select Add to-be-undeployed Object or Import Based on Lineage to add tasks that you want to undeploy.
Add to-be-undeployed Object: Add tasks that you want to undeploy by specifying the IDs of the tasks.
Import Based on Lineage: Select a task and select tasks based on lineage levels.
NoteYou can view the ID of the node task in the General section of the Properties panel on the task configuration page.
You can also go to Operation Center and choose in the left-side navigation pane to view the ID of the task node.
Undeploy tables: You can select Add to-be-undeployed Object to undeploy tables.
You cannot select an object that is already added to an undeployment plan. Each object can be added only to one undeployment plan.
Specific types of objects cannot be added to an undeployment plan. For example, you cannot add the root node of a workspace to an undeployment plan or add a zero load node that you created for an undeployment plan to the undeployment plan.
Information about objects to be undeployed
Item | Description |
Undeployment Impact Assessment |
|
Impact Item Description | The description of the undeployment impact assessment. |
The output table is consumed. | Information about the consumption of the output table of the task. |
Task That Generates Table | The output task of the table. You can click the number of tasks to view the task information. |
Number of Affected Baselines/Highest Priority |
Note For more information about task baseline monitoring, see Manage baselines. |
Step 3: Configure undeployment settings
After you add objects that you want to undeploy, you can configure fine-grained undeployment settings, including the Start Undeployment, Delayed Scheduling, Suspend Scheduling, Undeploy Task, Rename Output Table, Reset Table Lifecycle, Delete Output Table, and Undeploy Completed phases.
Phase | Description | Scenario |
Start Undeployment | Start the graceful undeployment. This phase is selected by default and cannot be unselected. | |
Delayed Scheduling | Delay the scheduling time of the objects that you want to undeploy. In this case, you delay the time when the output tables of the objects and their descendant objects are generated. | If no data is generated by the ancestor platform, you can select this phase to ensure that the tasks can run as normal. |
Suspend Scheduling | Freeze the objects that you want to undeploy. To freeze the objects, go to Operation Center. In the left-side navigation pane, choose . On the page that appears, find the objects that you want to freeze and freeze the objects. The objects and their descendant objects no longer generate tables. | You can select this phase if you have suspended a business but want to keep the workflow. |
Undeploy Task | Undeploy the desired objects. After the objects are undeplyed, no instance is generated on the next day. The scheduling mode of the existing instances is set to dry run and the system no longer runs these instances. This phase is selected by default and you can unselect it. | You can select this phase to undeploy a node or workflow that is no longer needed. |
Rename Output Table | Rename the output table of a task or table. | You can select this phase if you want to identify the output table of an undeployed object. |
Reset Table Lifecycle | Reset the lifecycle of the output table of a task or table. | You can select this phase if you want to retain the output table of an undeployed object for a period of time. Note Only MaxCompute tables are supported. |
Delete Output Table | Delete the output table of a task or table. | You can select this phase you do not need the output table of an undeployed object. |
Undeploy Completed | End the graceful undeployment. This phase is selected by default and cannot be canceled. |
The steps in the phases of the graceful undeployment plan are performed in sequence. You can specify whether to perform the step in each phase. If you unselect a phase, the phase is skipped by default and the system proceeds to the next phase.
The graceful undeployment plan allows you to send notifications to the specified contacts by internal messages, emails, and DingTalk chatbots. For more information about how to configure a DingTalk chatbot, see the "Scenario practices: Send alert notifications to a DingTalk group" section of the Create a custom alert rule topic. For information about system prompt messages, go to the Data Governance Center page and click Workbench. In the left-side navigation pane, click Governance Reminder in the To-do List section.
Step 4: Specify the method to execute the undeployment plan
You can specify the method to perform the step in each phase of the undeployment plan. You can select Manually Trigger Execution or Automatic Execution for the Execution Mode parameter.
Manually Trigger Execution
Note: Each step of the undeployment process can be performed only after the previous step is confirmed by the creator of the undeployment plan.
Recommended scenario: The task or table is used in important situations of the business workflow.
Automatic Execution
Note: Each step of the undeployment process is automatically performed based on the silence period that you configured. A step is automatically performed when the silence period of the previous step ends.
Recommended scenario: You need to undeploy a large number of business workflows and the importance of the workflows is low.
Step 5: View and manage the undeployment plan
On the Graceful Shutdown page, you can view the list of undeployment plans. You can execute, cancel, or delete an undeployment plan.
If you cancel an undeployment plan, the system cancels undeployment for tasks that are in the Undeploying state in the undeployment plan. Undeployed tasks are not affected.
If you delete an undeployment plan, the system cancels undeployment for tasks that are in the Undeploying state in the undeployment plan. Undeployed tasks are not affected. After the undeployment is canceled, the system deletes the undeployment plan. The deletion operation is irreversible.
You can click Execution Details in the Actions column of an undeployment plan to view the configuration details of the plan. You can also perform the operations that are described in the following table on a single task.
Operation | Description |
View undeployment details | You can view the undeployment details of a single task, such as the undeployment progress, task status, and execution details of each phase. You can also perform various operations on the task, such as canceling undeployment for the task and rolling back the undeployment operation for the task.
|
Go to Operation Center | You can go to Operation Center in the production environment to view the O&M details of the task. For more information, see Overview. |
Go to the configuration tab of the task | You can go to the configuration tab of the task in DataStudio to edit the task. For more information, see Overview. |
Appendix: Best practices for graceful undeployment
Graceful undeployment provides an easier way to undeploy tasks. In this section, the Big Data Experience, ETL workflow Template, and Website User Behavior Analysis workflows of DataWorks are used as examples to demonstrate how to gracefully undeploy tasks and tables.
Prerequisites
A DataWorks workspace in standard mode is created.
A MaxCompute data source is bound to the workspace.
Step 1: Create a graceful undeployment plan
Select a target.
Undeploy the Website User Behavior Analysis workflow and delete the output tables in the production environment.
Go to the graceful undeployment page.
In the upper-left corner of the DataStudio page, click the
icon and choose . On the Data Asset Governance page, choose in the left-side navigation pane to go to the Graceful Shutdown page.
Configure basic graceful undeployment settings.
Click Create Plan to go to the configuration page. The following table describes the parameters.
Section
Parameter
Description
Basic Information
Workspace
Select the workspace of the Website User Behavior Analysis workflow or the workspace of the tasks that you want to undeploy.
NoteOnly workspaces in which you are assigned the Workspace Administrator and O&M roles are displayed.
Plan Name
Specify a name for the graceful undeployment plan.
Plan Description
Enter the description of the graceful undeployment plan.
To-Be-Undeployed Object
Task
Add to-be-undeployed Object
Click Add to-be-undeployed Object. In the Add Task dialog box, search for tasks based on task IDs.
Import Based on Lineage
Click Import Based on Lineage. In the Add Task dialog box, search for tasks based on the following content.
Task: Select the tasks that you want to undeploy. In this example, you need to undeploy the Website User Behavior Analysis workflow. Therefore, select the root node task of the task.
Lineage Type: Select Task Lineage.
Selected Level: Select the current level of the task.
NoteThe highest level is level 6.
After you complete the configuration, click Search to find all desired tasks. After you confirm the tasks, click Confirmation to save the selection.
Table
Add to-be-undeployed Object
Data Source Type: Select MaxCompute.
Project: Specify the MaxCompute project bound to the production environment.
Table Name: Specify the name of the table in the production environment.
Owner: Only members in the workspaces in which you are assigned the Worksapce Administrator role are displayed. Enter keywords to search for more users.
Output Table: Specify whether the selected table is an output table.
Baseline Task: Enter the baseline level. A value of 0 indicates that no baseline is configured.
Data Source Type: Select E-MapReduce.
Cluster: Specify the E-MapReduce (EMR) cluster bound to the production environment.
Database: Specify a database in the cluster.
Table Name: Specify the name of the table in the production environment.
Owner: Only members in the workspaces in which you are assigned the Worksapce Administrator role are displayed. Enter keywords to search for more users.
Output Table: Specify whether the selected table is an output table.
In this example, select Task, click Import Based on Lineage and then select Automatic Execution to undeploy tasks, or select Table, click Add to-be-undeployed Object, and then select Automatic Execution to undeploy tables.
Add tasks and tables that you want to undeploy
In the To-Be-Undeployed Object section, select Task and click Import Based on Lineage. In the Add Task dialog box, select Website User Behavior Analysis from the Task drop-down list and click Confirmation. Then, click OK to proceed to the next step.
Parameter
Description
Task
Select the zero load ancestor node named Case for User Behavior Analysis of the Website User Behavior Analysis workflow.
Selected Level
Select level 6 to find all nodes of the Website User Behavior Analysis workflow.
Select the nodes of the tasks that you want to undeploy and click OK to add the nodes.
In the To-Be-Undeployed Object section, select Table and click Add to-be-undeployed Object. In the Add Table dialog box, select Website User Behavior Analysis from the Project drop-down list and click Confirmation. Then, click OK to proceed to the next step.
In this example, the table named
ods_user_info_din the Website User Behavior Analysis workflow is specified.Parameter
Description
Data Source Type
Select MaxCompute.
Project
Specify the MaxCompute project bound to the DataWorks workspace.
Table Name
Specify the
ods_user_info_dtable in the Website User Behavior Analysis workflow.NoteSeparate multiple table names with commas (
,).After you click Search, the
ods_user_info_dtable in the production environment is displayed. Select the table and click Confirmation.
Configure graceful undeployment settings.
After you add objects that you want to undeploy in the Add to-be-undeployed Object section, you can configure graceful undeployment settings. You can configure fine-grained undeployment settings, including the Start Undeployment, Delayed Scheduling, Suspend Scheduling, Undeploy Task, Rename Output Table, Reset Table Lifecycle, Delete Output Table, and Undeploy Completed phases. You can customize the configuration on demand. For more information about the required parameters, see the "Step 3: Configure undeployment settings" section of this topic.
Start Undeployment is selected by default to start the graceful deployment.
Select Undeploy Task and specify the Send Notification, Notification Recipient, and Silence Period parameters.
Select Reset Table Lifecycle, configure the lifecycle of the reset table, and specify the Send Notification, Notification Recipient, and Silence Period parameters.
Select Delete Output Table and specify the Send Notification and Notification Recipient parameters.
Undeploy Completed is selected by default.
NoteUndeploy Task: This phase is selected by default. When you choose to undeploy a table, the node task of the output table is undeployed.
Reset Table Lifecycle: This option is available only when you undeploy MaxCompute tables. This option is unavailable for EMR tables.
Silence Period:
Note: The duration between the time when the step in the current phase is performed to the time when you can perform the next step. For example, you specify a silence period for the scheduling suspension step of an object that you want to undeploy. In this case, you can undeploy the object only after the silence period expires.
Recommended scenario: Configure this parameter when you want to send notifications to the specified contacts during each phase of the undeployment process.
Specify the method to execute the undeployment plan.
You can use the following methods to execute the undeployment plan. In this example, Automatic Execution is selected.
Manually Trigger Execution: The step in each phase of the node undeployment process must be confirmed by the creator of the node undeployment plan before the next step can be performed.
Automatic Execution: The step in each phase of the node undeployment process starts to be performed after the configured silence period expires. Then, you are navigated to the step in the next phase.
Complete the creation.
You can use the following methods to complete the creation. In this example, Create is selected.
Create: Create the graceful undeployment plan and redirect to the Running Details page. Then, manually select the tasks that you want to undeploy and click Batch Execute below the list to execute the plan.
Create and Execute: Create the graceful undeployment plan and redirect to the Running Details page to execute the plan.
Step 2: View the execution details of the graceful undeployment plan
Go to the graceful undeployment page.
In the upper-left corner of the DataStudio page, click the
icon and choose . In the top navigation bar of the Data Governance Center page, click Workbench. The Governance Issue page appears. In the left-side navigation pane, click Graceful Shutdown to go to the Graceful Shutdown page.
View the execution details of the undeployment plan.
You can query undeployment plans based on the Plan Name, Creator, Plan Status, Governance Object Type, and Plans I Created conditions.
Find the graceful undeployment plan for the Website User Behavior Analysis workflow. The Actions column displays the following operations.
Running Details: Redirect to the execution details of the plan.
Execute: Execute the plan.
Cancel: Cancel the plan.
Delete: Delete the plan.
Click Running Details in the Actions column of the undeployment plan for the Website User Behavior Analysis workflow to redirect to the details page of the plan.
Execution details
On the details page, you can view Plan Information, Undeployment Process, and To-Be-Undeployed Object of the undeployment plan for the Website User Behavior Analysis workflow.
Plan Information: displays the Workspace, Creator, Creation Time, Undeployment Type, Number of Tasks, Undeployment Status, Process Execution Mode, and Snapshot Time parameters of the plan.
Undeployment Process: displays the number of tasks in each phase of the plan.
To-Be-Undeployed Object:
Displays the following information about the tasks to be undeployed: No., Task ID, Task Name, Impact Item Description, Number of Output Tables, Execution Progress, Executed Step, Step Status, Owner, Number of Affected Baselines/Highest Priority, Level-1 Descendant Tasks, Normally Scheduled Descendant Tasks, Cross-workspace Descendant Task, Task Type, and Overall Status.
The Actions column displays the following operations that you can perform on a single task.
Running Details: Redirect to the details page of the task and view the details of each step. You can also roll back or cancel the task, retry a step, or skip the current step.
Go to O&M: Go to the production environment O&M page of the task.
NoteThe production environment O&M page is no longer displayed for a deployed task.
When you choose to undeploy a table, the Go to O&M button is unavailable.
Cancel: Cancel the undeployment for the task.
