DataWorks:Best practices for calling DataWorks API operations to manage files

Procedure

1. Initialize a client.

   public class IdePopApiTest {
       // The account information.
       @Value("${aliyun.akId}")
       private String akId;
       @Value("${aliyun.akSecret}")
       private String akSecret ;
       
       // Example: cn-shanghai
       @Value("${aliyun.region}")
       private String region;
       // Example: dataworks.cn-shanghai.aliyuncs.com
       @Value("${aliyun.endpoint}")
       private String endpoint;
   
       private IAcsClient client;
   
   
       @Before
       public void init() throws ClientException {
           String product = "dataworks-public";
           IClientProfile profile = DefaultProfile.getProfile(region, akId, akSecret);
           DefaultProfile.addEndpoint(region, product, endpoint);
           client = new DefaultAcsClient(profile);
       }
   }   

2. Create a workflow.

   public void createBusiness() throws ClientException {
           CreateBusinessRequest request = new CreateBusinessRequest();
           // The ID of the workspace. You can view the ID on the Workspace Management page after you click Workspace Manage in the upper-right corner of the DataStudio page.
           request.setProjectId(78837L);
           // The name of the workspace. You can view the name on the Workspace Management page after you click Workspace Manage in the upper-right corner of the DataStudio page.
           request.setProjectIdentifier("zxy_8221431");
           // The unique name of the workflow.
           request.setBusinessName("test_create_business");
           // NORMAL indicates an auto triggered workflow and MANUAL_BIZ indicates a manually triggered workflow.
           request.setUseType("NORMAL");
           CreateBusinessResponse acsResponse = client.getAcsResponse(request);
           log.info(acsResponse.toString());
   }

   Note

   • You can call the UpdateBusiness operation to modify a workflow and call the DeleteBusiness operation to delete a workflow. Before you delete a workflow, make sure that all nodes in the workflow are deleted.

   • You can call the ListBusiness operation to query the workflows that meet the filter conditions in a specific workspace.

   • You can call the GetBusiness operation to query the details of a workflow.

3. Create a folder.

   public void createFloder() throws ClientException {
           CreateFolderRequest request = new CreateFolderRequest();
           request.setProjectId(78837L);
           // Path: "Workflow/Name of the desired workflow/Name of the folder/Name of the latest folder"
           request.setFolderPath("Workflow/test_create_business/Common/test_floder_123");
           CreateFolderResponse acsResponse = client.getAcsResponse(request);
           log.info(acsResponse.toString());
       }

   Note

   You can call the UpdateFolder operation to change the name of a folder. You can call the DeleteFolder operation to delete a folder from the Scheduled Workflow pane. Before you delete a folder from the Scheduled Workflow pane, make sure that all nodes in the folder are deleted.

4. Create, commit, and deploy a node.

   1. Create a node.

      public void createFile() throws ClientException{
              CreateFileRequest request = new CreateFileRequest();
              request.setProjectId(78837L);
              request.setProjectIdentifier("zxy_8221431");
              request.setFileName("test_vi_file_name2");
              request.setFileDescription("Description");
              request.setFileFolderPath("Workflow/test_create_business/Common/test_floder_123");
              // The type of the node. The node type must match the folder type. For example, a batch synchronization node for which the value of the fileType parameter is set to 23 must be stored in the Data Integration folder. 
              request.setFileType(99);
              // The output of the file on which the current file depends. In this topic, the ancestor file is the file for the root node and the output of the ancestor file is projectIdentifier_root. This parameter is required. If this parameter is left empty, the node cannot be committed. 
              request.setInputList("zxy_8221431_root");
              // The rerun type that corresponds to the Rerun parameter in the Schedule section on the Properties tab of the DataStudio page. This parameter is required. If this parameter is left empty, the node cannot be committed. 
              // ALL_ALLOWED: The node can be rerun regardless of whether it is successfully run or fails to run.
              // FAILURE_ALLOWED: The node can be rerun only after it fails to run.
              // ALL_DENIED: The node cannot be rerun regardless of whether it is successfully run or fails to run.
              request.setRerunMode("ALL_ALLOWED");
              CreateFileResponse acsResponse = client.getAcsResponse(request);
              // acsResponse.getData() indicates the node ID, which is required when you commit the node.
              Long fileId = acsResponse.getData();
              log.info(acsResponse.toString());
          }

      Note

      • The node type must match the folder type. For example, a batch synchronization node for which the value of the fileType parameter is set to 23 must be stored in the Data Integration folder.

      • You can call the ListFile operation to query the desired file. You can call the GetFile operation to query the details of a file.

      • When you call the UpdateFile operation to update an existing file, the settings of the parameters that you configure for the file must be different from the original settings of the same parameters for the file. For example, if the original value of a parameter is A, you must change the value of the parameter to B before you commit the node. If you set the parameter to A, an exception that indicates invalid data occurs.

   2. Commit the node.

       public void submitFile() throws ClientException{
              SubmitFileRequest request = new SubmitFileRequest();
              request.setProjectId(78837L);
              request.setProjectIdentifier("zxy_8221431");
              // The node ID that is returned when the node is created. The node ID corresponds to the value of the file_id field in a file table in a database. The node ID is not the ID that is displayed in the General section on the Properties tab of the DataStudio page. 
              request.setFileId(501576542L);
              request.setComment("Comment");
              SubmitFileResponse acsResponse = client.getAcsResponse(request);
              // Call the GetDeployment operation to obtain the details of the related deployment package. 
              Long deploymentId = acsResponse.getData();
              log.info(acsResponse.toString());
      }

      Note

      If you use a workspace in standard mode, you must deploy the node to the production environment. If you use a workspace in standard mode, you must deploy the operation of deleting a node to the production environment to make the operation take effect. After you commit or deploy a node, you can call the GetDeployment operation to query the details of the deployment package.

   3. Deploy the node.

       public void deploy() throws ClientException{
              DeployFileRequest request = new DeployFileRequest();
              // request.setProjectId(78837L);
              request.setProjectIdentifier("zxy_8221431");
              request.setFileId(501576542L);
              request.setComment("Comment");
              // Select this parameter or the file_id parameter. The value of the NodeId parameter is the value of the Node ID parameter in the General section on the Properties tab of the DataStudio page. 
              request.setNodeId(700004537241L);
              DeployFileResponse acsResponse = client.getAcsResponse(request);
              // Call the GetDeployment operation to obtain the details of the related deployment package. 
              Long deploymentId = acsResponse.getData();
              log.info(acsResponse.getData().toString());
          }

      Note

      If you use a workspace in basic mode, the development and production environments are not isolated. You need to only call the API operation that is used to commit nodes.

5. Obtain the details of the deployment package.

   public void getDeployment() throws ClientException{
           GetDeploymentRequest request = new GetDeploymentRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
           // The deployment package ID that is returned after you commit or deploy the node. 
           request.setDeploymentId(2776067L);
           GetDeploymentResponse acsResponse = client.getAcsResponse(request);
           log.info(acsResponse.getData().toString());
       }

   Note

   When you commit or deploy a node, the deployment package ID is returned by a response parameter. You can call this API operation to query the details of the deployment package. If 1 is returned, the node is deployed. For information about the descriptions of response parameters, see GetDeployment.

6. Delete the node.

   public void deleteFile() throws ClientException{
           DeleteFileRequest request = new DeleteFileRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
   
           request.setFileId(501576542L);
           DeleteFileResponse acsResponse = client.getAcsResponse(request);
           log.info(acsResponse.toString());
           // If you use a workspace in standard mode, you must deploy the operation of deleting a node to the production environment to make the operation take effect. 
           DeployFileRequest deployFileRequest = new DeployFileRequest();
           request.setProjectId(78837L);
           deployFileRequest.setProjectIdentifier("zxy_8221431");
           deployFileRequest.setFileId(501576542L);
           deployFileRequest.setComment("Comment");
           // Select this parameter or the file_id parameter. The value of the NodeId parameter is the value of the Node ID parameter in the General section on the Properties tab of the DataStudio page. 
           deployFileRequest.setNodeId(700004537241L);
           DeployFileResponse deployFileResponse = client.getAcsResponse(deployFileRequest);
           Long deploymentId = deployFileResponse.getData();
           log.info(deployFileResponse.getData().toString());
   
       }

   Note

   If you use a workspace in standard mode, you must call the DeployFileRequest operation to deploy the operation of deleting a node to the production environment to make the operation take effect.

7. Delete the folder.

   You must delete all nodes and subfolders in a folder before you delete the folder. You can call the ListFilesRequest operation to query all nodes in a folder. Then, delete the nodes and the folder. You can call the following API operations in sequence:

   1. ListFolders: Queries the ID of the folder that you want to delete and queries the directory of the folder.

   2. ListFiles: Queries the information about all nodes in the folder.

   3. DeleteFile: Deletes all nodes in the folder. If you use a workspace in standard mode, you also need to call the deploy operation to deploy the delete operation to the production environment.

   4. DeleteBusiness: Deletes the workflow.

   public void deleteFolder() throws ClientException{
           // Delete all nodes in the folder before you delete the folder. Call the ListFiles operation to query nodes. 
           List<ListFilesResponse.Data.File> files = listFiles();
           for (ListFilesResponse.Data.File file : files) {
               // Delete all nodes in the folder. 
               deleteFile(file);
           }
           DeleteFolderRequest request = new DeleteFolderRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
           // Call the ListFolders operation to query the ID of the folder. 
           request.setFolderId("*****");
           DeleteFolderResponse acsResponse = client.getAcsResponse(request);
           log.info(acsResponse.toString());
    }
   
   /**
        * Query nodes.
        * @return
        * @throws ClientException
        */
   public List<ListFilesResponse.Data.File> listFiles() throws ClientException{
           ListFilesRequest request = new ListFilesRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
           request.setPageNumber(1);
           request.setPageSize(10);
           request.setFileFolderPath("Workflow/test_create_business/Common/test_floder_123");
           // Match keywords. 
           //request.setKeyword("");
           // The functional module to which the file belongs. Valid values: NORMAL, MANUAL, MANUAL_BIZ, SKIP, ADHOCQUERY, and COMPONENT. The value NORMAL indicates an auto triggered workflow. The value MANUAL indicates a manually triggered node. The value MANUAL_BIZ indicates a manually triggered workflow. The value SKIP indicates a dry-run DataStudio node. The value ADHOCQUERY indicates an ad hoc query. The value COMPONENT indicates snippets.
           request.setUseType("NORMAL");
           // The type of the node. Separate multiple node types with commas (,). 
           request.setFileTypes("10,23,99");
           ListFilesResponse acsResponse = client.getAcsResponse(request);
           return acsResponse.getData().getFiles();
       }
   
   /**
        * Query folders.
        * @return
        * @throws ClientException
        */
    public List<ListFoldersResponse.Data.FoldersItem> listFolders() throws ClientException{
           ListFoldersRequest request = new ListFoldersRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
           request.setPageNumber(1);
           request.setPageSize(10);
           // The path of the parent folder. 
           request.setParentFolderPath("Workflow/test_create_business/Common");
           ListFoldersResponse acsResponse = client.getAcsResponse(request);
           return acsResponse.getData().getFolders();
       }

8. Delete the workflow.

   You must delete all nodes in a workflow before you delete the workflow. You can call the following API operations in sequence:

   1. ListFiles: Queries the information about all nodes in the workflow.

   2. DeleteFile: Deletes all nodes. If you use a workspace in standard mode, you also need to call the deploy operation to deploy the delete operation to the production environment.

   3. DeleteBusiness: Deletes the workflow.

   public void deleteBusiness() throws ClientException{
           // You must delete all nodes in a workflow before you delete the workflow. 
           DeleteBusinessRequest request = new DeleteBusinessRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
           // Call the ListBusiness operation to query the ID of the desired workflow. 
           List<ListBusinessResponse.Data.BusinessItem> businessItems = listBusiness();
   
           request.setBusinessId(businessItems.get(0).getBusinessId());
           DeleteBusinessResponse acsResponse = client.getAcsResponse(request);
           log.info(acsResponse.toString());
   
       }
   
       public List<ListBusinessResponse.Data.BusinessItem> listBusiness()  throws ClientException{
           ListBusinessRequest request = new ListBusinessRequest();
           request.setProjectId(78837L);
           request.setProjectIdentifier("zxy_8221431");
           request.setPageNumber(1);
           request.setPageSize(10);
           request.setKeyword("test_create");
           ListBusinessResponse acsResponse = client.getAcsResponse(request);
           return acsResponse.getData().getBusiness();
       }