DataWorks:Best practice: Table Management OpenAPI Basic Practice

Practice 1: Query a list of metatables

This section describes how to call an API operation to query a list of metatables in a MaxCompute project and query data in the metatables by page.

1. Write the GetMetaDBTableList method in meta-service-proxy, process frontend parameters, and then send requests to the GetMetaDBTableList operation in the OpenAPI module to obtain the list of metatables.

   package com.aliyun.dataworks.services;
   
   import com.aliyun.dataworks.dto.*;
   import com.aliyuncs.IAcsClient;
   import com.aliyuncs.dataworks_public.model.v20200518.*;
   import com.aliyuncs.exceptions.ClientException;
   import com.aliyuncs.exceptions.ServerException;
   import com.aliyuncs.utils.StringUtils;
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.stereotype.Service;
   import org.springframework.util.CollectionUtils;
   
   /**
    * @author dataworks demo
    */
   @Service
   public class MetaServiceProxy {
     @Autowired
     private DataWorksOpenApiClient dataWorksOpenApiClient;
     /**
      * DataWorks OpenAPI : GetMetaDBTableList
      * Link : https://www.alibabacloud.com/help/doc-detail/173916.html
      *
      * @param listTablesDto
      */
     public GetMetaDBTableListResponse.Data getMetaDBTableList(ListTablesDto listTablesDto) {
       try {
         IAcsClient client = dataWorksOpenApiClient.createClient();
         GetMetaDBTableListRequest getMetaDBTableListRequest = new GetMetaDBTableListRequest();
         if (StringUtils.isEmpty(listTablesDto.getDataSourceType())) {
           // The type of the data source. Valid values: odps and emr. 
           getMetaDBTableListRequest.setDataSourceType("odps");
         }
         // The GUID of the MaxCompute project. Specify the GUID in the odps.{projectName} format. This parameter is required only if you set the DataSourceType parameter to odps. 
         getMetaDBTableListRequest.setAppGuid(listTablesDto.getAppGuid());
         // The number of the page to return. 
         getMetaDBTableListRequest.setPageNumber(listTablesDto.getPageNumber());
         // The number of entries to return on each page. Default value: 10. Maximum value: 100. 
         getMetaDBTableListRequest.setPageSize(listTablesDto.getPageSize());
         // The name of the database. 
         getMetaDBTableListRequest.setDatabaseName(listTablesDto.getDatabaseName());
         GetMetaDBTableListResponse acsResponse = client.getAcsResponse(getMetaDBTableListRequest);
         // The total number of compute engine instances.
         System.out.println(acsResponse.getData().getTotalCount());
         for (GetMetaDBTableListResponse.Data.TableEntityListItem tableEntityListItem : acsResponse.getData()
           .getTableEntityList()) {
           // The GUID of the metatable.
           System.out.println(tableEntityListItem.getTableGuid());
           // The name of the metatable.
           System.out.println(tableEntityListItem.getTableName());
           // The name of the database.
           System.out.println(tableEntityListItem.getDatabaseName());
         }
         return acsResponse.getData();
       } catch (ServerException e) {
         e.printStackTrace();
       } catch (ClientException e) {
         e.printStackTrace();
         // The ID of the request.
         System.out.println(e.getRequestId());
         // The error code.
         System.out.println(e.getErrCode());
         // The error message.
         System.out.println(e.getErrMsg());
       }
       return null;
     }
   }

2. Add the listMetaDBTable method to MetaRestController as an entrance that allows access from frontend clients.

   package com.aliyun.dataworks.demo;
   
   import com.aliyun.dataworks.dto.*;
   import com.aliyun.dataworks.services.MetaServiceProxy;
   import com.aliyuncs.dataworks_public.model.v20200518.*;
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.web.bind.annotation.GetMapping;
   import org.springframework.web.bind.annotation.PostMapping;
   import org.springframework.web.bind.annotation.RequestMapping;
   import org.springframework.web.bind.annotation.RestController;
   import org.springframework.web.bind.annotation.CrossOrigin;
   
   
   /**
    * Demonstrate how to build a custom metadata platform by using the OpenAPI module.
    *
    * @author dataworks demo
    */
   @RestController
   @RequestMapping("/meta")
   public class MetaRestController {
     @Autowired
     private MetaServiceProxy metaService;
     /**
      * Query data in the metatables by page.
      *
      * @param listTablesDto
      * @return {@link GetMetaDBTableListResponse.Data}
      */
     @GetMapping("/listTables")
     public GetMetaDBTableListResponse.Data listMetaDBTable(ListTablesDto listTablesDto) {
         return metaService.getMetaDBTableList(listTablesDto);
     }
   }

3. Provide a simple interactive interface on the frontend.

   Note

   The following sample code shows a simplified procedure of table queries and is a minimal implementation of table queries.

   import React from 'react';
   import { Table, Form, Field, Input, Button, Pagination, Dialog, Card } from '@alifd/next';
   
   function getTableList(params: TableListInput): Promise<TableListOutput> {
     const url = helpers.createFetchUrl(METHOD_URL.GET_TABLE_LIST, params);
     const response = await fetch(url);
     const result = await response.json();
     return result;
   }
   
   const { Column } = Table;
   const { Item } = Form;
   const { Header, Content } = Card;
   const App: React.FunctionComponent<Props> = () => {
     const field = Field.useField();
     const [datasource, setDatasource] = React.useState<TableItem[]>([]);
     const [loading, setLoading] = React.useState<boolean>(false);
     const [total, setTotal] = React.useState<number>(0);
     React.useEffect(() => field.setValue('dataSourceType', 'odps'), []);
     const onSubmit = React.useCallback((pageNumber: number = 1) => {
       field.validate(async (errors, values) => {
         if (errors) return;
         setLoading(true);
         try {
           const response = await getTableList({ pageNumber, ...values } as TableListInput);
           setDatasource(response.tableEntityList);
           setTotal(response.totalCount);
         } catch (e) {
           throw e;
         } finally {
           setLoading(false);
         }
       });
     }, [field]);
     return (
       <div>
         <Card free hasBorder={false}>
           <Header title="Demo for metatable management" />
           <Content style={{ marginTop: 24 }}>
             <Form field={field} colon fullWidth>
               <Item label="MaxCompute project GUID" name="appGuid" required>
                 <Input />
               </Item>
               <Item label="Database Name" name="databaseName">
                 <Input />
               </Item>
               <div>
                 <Button type="primary" onClick={() => onSubmit()}>Query</Button>
                 <Button type="primary">Create Table</Button>
               </div>
             </Form>
             <div>
               <Table dataSource={datasource} loading={loading}>
                 <Column title="Database Name" dataIndex="databaseName" />
                 <Column title="Table GUID" dataIndex="tableGuid" />
                 <Column title="Table Name" dataIndex="tableName" />
                 <Column title="Operation" width={150} cell={(value, index, record) => (
                   <div>
                     <Button type="primary" text>View Details</Button>
                     <Button type="primary" text>View Table Lineage</Button>
                   </div>
                 )} />
               </Table>
               <Pagination
                 total={total}
                 onChange={onSubmit}
                 showJump={false}
               />
             </div>
           </Content>
         </Card>
       </div>
     );
   };
   
   export default App;
                           

4. After you write the preceding code, you can deploy and run the project code offline. For information about how to deploy and run the project code, see Common operations: Deploy and run the code offline.

Practice 2: Query the details of a metatable

This section describes how to call the GetMetaTableBasicInfo, GetMetaTableColumn, and GetMetaTablePartition operations in the OpenAPI module to query the details of a metatable.

1. Write the getMetaTableBasicInfo, getMetaTableColumn, and getMetaTablePartition methods in meta-service-proxy to call the matched API operations in the OpenAPI module to obtain the details of a metatable, including the basic information, table fields, and table partitions.

   package com.aliyun.dataworks.services;
   
   import com.aliyun.dataworks.dto.*;
   import com.aliyuncs.IAcsClient;
   import com.aliyuncs.dataworks_public.model.v20200518.*;
   import com.aliyuncs.exceptions.ClientException;
   import com.aliyuncs.exceptions.ServerException;
   import com.aliyuncs.utils.StringUtils;
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.stereotype.Service;
   import org.springframework.util.CollectionUtils;
   
   /**
    * @author dataworks demo
    */
   @Service
   public class MetaServiceProxy {
     @Autowired
     private DataWorksOpenApiClient dataWorksOpenApiClient;
     /**
      * DataWorks OpenAPI : GetMetaTableBasicInfo
      * Link : https://www.alibabacloud.com/help/doc-detail/173920.html
      *
      * @param getMetaTableBasicInfoDto
      */
     public GetMetaTableBasicInfoResponse.Data getMetaTableBasicInfo(GetMetaTableBasicInfoDto getMetaTableBasicInfoDto) {
       try {
         IAcsClient client = dataWorksOpenApiClient.createClient();
         GetMetaTableBasicInfoRequest getMetaTableBasicInfoRequest = new GetMetaTableBasicInfoRequest();
         // The GUID of the MaxCompute metatable. Specify the GUID in the odps.projectName.tableName format. 
         getMetaTableBasicInfoRequest.setTableGuid(getMetaTableBasicInfoDto.getTableGuid());
         // The name of the database.
         getMetaTableBasicInfoRequest.setDatabaseName(getMetaTableBasicInfoDto.getDatabaseName());
         // The name of the metatable in the E-MapReduce (EMR) cluster.
         getMetaTableBasicInfoRequest.setTableName(getMetaTableBasicInfoDto.getTableName());
         // The type of the data source. Valid values: odps and emr. 
         getMetaTableBasicInfoRequest.setDataSourceType("odps");
         // Specifies whether to include extended fields in query results. The extended fields include ReadCount, FavoriteCount, and ViewCount. This parameter takes effect only if you set the DataSourceType parameter to odps. 
         getMetaTableBasicInfoRequest.setExtension(true);
         GetMetaTableBasicInfoResponse acsResponse = client.getAcsResponse(getMetaTableBasicInfoRequest);
         // The name of the metatable. 
         System.out.println(acsResponse.getData().getTableName());
         // The number of times that the metatable was added to favorites. This parameter takes effect only if you set the DataSourceType parameter to odps. The value of this parameter is returned only if you set the Extension parameter to true. 
         System.out.println(acsResponse.getData().getFavoriteCount());
         // The description of the metatable. 
         System.out.println(acsResponse.getData().getComment());
         // The number of fields. 
         System.out.println(acsResponse.getData().getColumnCount());
         // The time when the metatable was created. 
         System.out.println(acsResponse.getData().getCreateTime());
         // The ID of the workspace. 
         System.out.println(acsResponse.getData().getProjectId());
         // The ID of the metatable owner. 
         System.out.println(acsResponse.getData().getOwnerId());
         // The type of the environment. Valid values:
         // 0: The metatable is in the development environment. 
         // 1: The metatable is in the production environment. 
         System.out.println(acsResponse.getData().getEnvType());
         // The name of the database. 
         System.out.println(acsResponse.getData().getDatabaseName());
         // The visibility of the metatable. Valid values: 0: The metatable is visible to members of the workspace. 
         // 1: The metatable is visible to users within the tenant. 
         // 2: The metatable is visible to all tenants. 
         // 3: The metatable is visible only to the metatable owner. 
         System.out.println(acsResponse.getData().getIsVisible());
         // The GUID of the metatable. 
         System.out.println(acsResponse.getData().getTableGuid());
         // The number of times that the metatable was read. This parameter takes effect only if you set the DataSourceType parameter to odps. The value of this parameter is returned only if you set the Extension parameter to true. 
         System.out.println(acsResponse.getData().getReadCount());
         // The ID of the EMR cluster. 
         System.out.println(acsResponse.getData().getClusterId());
         // Indicates whether the metatable is a partitioned table. Valid values:
         // true: The metatable is a partitioned table. 
         // false: The metatable is a non-partitioned table. 
         System.out.println(acsResponse.getData().getIsPartitionTable());
         // Indicates whether the metatable is a view. Valid values:
         // true: The metatable is a view. 
         // false: The metatable is not a view. 
         System.out.println(acsResponse.getData().getIsView());
         // The lifecycle of the metatable. Unit: days. 
         System.out.println(acsResponse.getData().getLifeCycle());
         // The name of the workspace. 
         System.out.println(acsResponse.getData().getProjectName());
         // The number of times that the metatable was viewed. This parameter takes effect only if you set the DataSourceType parameter to odps. The value of this parameter is returned only if you set the Extension parameter to true. 
         System.out.println(acsResponse.getData().getViewCount());
         // The time when the metatable was last accessed. 
         System.out.println(acsResponse.getData().getLastAccessTime());
         // The storage space that is occupied by the metatable. Unit: bytes. 
         System.out.println(acsResponse.getData().getDataSize());
         // The time when the metatable was last updated. 
         System.out.println(acsResponse.getData().getLastModifyTime());
         // The time when the schema of the metatable was last changed. 
         System.out.println(acsResponse.getData().getLastDdlTime());
         // The partition key of the Hive metatable. 
         System.out.println(acsResponse.getData().getPartitionKeys());
         // The storage path of the Hive metadatabase. 
         System.out.println(acsResponse.getData().getLocation());
         // The display name of the metatable. 
         System.out.println(acsResponse.getData().getCaption());
         // The ID of the tenant. 
         System.out.println(acsResponse.getData().getTenantId());
         return acsResponse.getData();
       } catch (ServerException e) {
         e.printStackTrace();
       } catch (ClientException e) {
         e.printStackTrace();
         // The ID of the request.
         System.out.println(e.getRequestId());
         // The error code.
         System.out.println(e.getErrCode());
         // The error message.
         System.out.println(e.getErrMsg());
       }
       return null;
     }
     /**
      * DataWorks OpenAPI : GetMetaTableColumn
      * Link : https://www.alibabacloud.com/help/doc-detail/173921.html
      *
      * @param getMetaTableColumnDto
      */
     public GetMetaTableColumnResponse.Data getMetaTableColumn(GetMetaTableColumnDto getMetaTableColumnDto) {
       try {
         IAcsClient client = dataWorksOpenApiClient.createClient();
         GetMetaTableColumnRequest getMetaTableColumnRequest = new GetMetaTableColumnRequest();
         // The GUID of the metatable. 
         getMetaTableColumnRequest.setTableGuid(getMetaTableColumnDto.getTableGuid());
         // The number of the page to return. 
         getMetaTableColumnRequest.setPageNum(getMetaTableColumnDto.getPageNum());
         // The number of entries to return on each page. Default value: 10. Maximum value: 100. 
         getMetaTableColumnRequest.setPageSize(getMetaTableColumnDto.getPageSize());
         // The ID of the EMR cluster. You can log on to the EMR console to obtain the ID. 
         getMetaTableColumnRequest.setClusterId(getMetaTableColumnDto.getClusterId());
         // The name of the metadatabase of the EMR cluster.
         getMetaTableColumnRequest.setDatabaseName(getMetaTableColumnDto.getDatabaseName());
         // The name of the metadatabase of the EMR cluster.
         getMetaTableColumnRequest.setTableName(getMetaTableColumnDto.getTableName());
         // The type of the data source. Only emr is supported. 
         getMetaTableColumnRequest.setDataSourceType(getMetaTableColumnDto.getDataSourceType());
         GetMetaTableColumnResponse acsResponse = client.getAcsResponse(getMetaTableColumnRequest);
         // The total number of fields. 
         System.out.println(acsResponse.getData().getTotalCount());
         for (GetMetaTableColumnResponse.Data.ColumnListItem columnListItem : acsResponse.getData()
           .getColumnList()) {
           // The GUID of the field. 
           System.out.println(columnListItem.getColumnGuid());
           // The name of the field. 
           System.out.println(columnListItem.getColumnName());
           // Indicates whether the field is a partition field. Valid values:
           // true: The field is a partition field. 
           // false: The field is not a partition field. 
           System.out.println(columnListItem.getIsPartitionColumn());
           // The comment for the field. 
           System.out.println(columnListItem.getComment());
           // The data type of the field. 
           System.out.println(columnListItem.getColumnType());
           // Indicates whether the field is the primary key. Valid values:
           // true: The field is the primary key. 
           // false: The field is not the primary key. 
           System.out.println(columnListItem.getIsPrimaryKey());
           // The ordinal number of the field. 
           System.out.println(columnListItem.getPosition());
           // The description of the field. 
           System.out.println(columnListItem.getCaption());
           // Indicates whether the field is a foreign key. Valid values:
           // true: The field is a foreign key. 
           // false: The field is not a foreign key. 
           System.out.println(columnListItem.getIsForeignKey());
           // The number of times that the field was read. 
           System.out.println(columnListItem.getRelationCount());
         }
         return acsResponse.getData();
       } catch (ServerException e) {
         e.printStackTrace();
       } catch (ClientException e) {
         e.printStackTrace();
         // The ID of the request.
         System.out.println(e.getRequestId());
         // The error code.
         System.out.println(e.getErrCode());
         // The error message.
         System.out.println(e.getErrMsg());
       }
       return null;
     }
       /**
      * DataWorks OpenAPI : GetMetaTablePartition
      * Link : https://www.alibabacloud.com/help/doc-detail/173923.html
      *
      * @param getMetaTablePartitionDto
      */
     public GetMetaTablePartitionResponse.Data getMetaTablePartition(GetMetaTablePartitionDto getMetaTablePartitionDto) {
       try {
         IAcsClient client = dataWorksOpenApiClient.createClient();
         GetMetaTablePartitionRequest getMetaTablePartitionRequest = new GetMetaTablePartitionRequest();
         // The number of the page to return. 
         getMetaTablePartitionRequest.setPageNumber(getMetaTablePartitionDto.getPageNumber());
         // The number of entries to return on each page. Default value: 10. Maximum value: 100. 
         getMetaTablePartitionRequest.setPageSize(getMetaTablePartitionDto.getPageSize());
         // The GUID of the metatable. 
         getMetaTablePartitionRequest.setTableGuid(getMetaTablePartitionDto.getTableGuid());
         // The ID of the EMR cluster. This parameter is required only if you set the DataSourceType parameter to emr. 
         getMetaTablePartitionRequest.setClusterId(getMetaTablePartitionDto.getClusterId());
         // The name of the database. This parameter is required only if you set the DataSourceType parameter to emr. 
         getMetaTablePartitionRequest.setDatabaseName(getMetaTablePartitionDto.getDatabaseName());
         // The name of the database. This parameter is required only if you set the DataSourceType parameter to emr. 
         getMetaTablePartitionRequest.setTableName(getMetaTablePartitionDto.getTableName());
         // The type of the data source. Valid values: odps and emr. 
         getMetaTablePartitionRequest.setDataSourceType(getMetaTablePartitionDto.getDataSourceType());
         GetMetaTablePartitionResponse acsResponse = client.getAcsResponse(getMetaTablePartitionRequest);
         for (GetMetaTablePartitionResponse.Data.DataEntityListItem dataEntityListItem : acsResponse.getData()
           .getDataEntityList()) {
           // The path of the partition. 
           System.out.println(dataEntityListItem.getPartitionPath());
           // The size of the partition. Unit: bytes. 
           System.out.println(dataEntityListItem.getDataSize());
           // The name of the partition. 
           System.out.println(dataEntityListItem.getPartitionName());
           // The comment. 
           System.out.println(dataEntityListItem.getComment());
           // The time when the partition was last modified. 
           System.out.println(dataEntityListItem.getModifiedTime());
           // The time when the partition was created. 
           System.out.println(dataEntityListItem.getCreateTime());
           // The number of entries in the partition. 
           System.out.println(dataEntityListItem.getRecordCount());
           // The type of the partition. 
           System.out.println(dataEntityListItem.getPartitionType());
           // The GUID of the partition. 
           System.out.println(dataEntityListItem.getPartitionGuid());
           // The location of the Hive partition. 
           System.out.println(dataEntityListItem.getPartitionLocation());
           // The GUID of the metatable. 
           System.out.println(dataEntityListItem.getTableGuid());
         }
         return acsResponse.getData();
       } catch (ServerException e) {
         e.printStackTrace();
       } catch (ClientException e) {
         e.printStackTrace();
         // The ID of the request.
         System.out.println(e.getRequestId());
         // The error code.
         System.out.println(e.getErrCode());
         // The error message.
         System.out.println(e.getErrMsg());
       }
       return null;
     }
   }

2. Add the getMetaTableBasicInfo, getMetaTableColumn, and getMetaTablePartition methods to MetaRestController as entries that allow access from frontend clients.

   package com.aliyun.dataworks.demo;
   
   import com.aliyun.dataworks.dto.*;
   import com.aliyun.dataworks.services.MetaServiceProxy;
   import com.aliyuncs.dataworks_public.model.v20200518.*;
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.web.bind.annotation.GetMapping;
   import org.springframework.web.bind.annotation.PostMapping;
   import org.springframework.web.bind.annotation.RequestMapping;
   import org.springframework.web.bind.annotation.RestController;
   import org.springframework.web.bind.annotation.CrossOrigin;
   
   
   /**
    * Demonstrate how to build a custom metadata platform by using the OpenAPI module.
    *
    * @author dataworks demo
    */
   @RestController
   @RequestMapping("/meta")
   public class MetaRestController {
     @Autowired
     private MetaServiceProxy metaService;
     /**
      * Obtain the basic information about the metatable.
      *
      * @param getMetaTableBasicInfoDto
      * @return {@link GetMetaTableBasicInfoResponse.Data}
      */
     @GetMapping("/getTable")
     public GetMetaTableBasicInfoResponse.Data getMetaTableBasicInfo(GetMetaTableBasicInfoDto getMetaTableBasicInfoDto) {
         return metaService.getMetaTableBasicInfo(getMetaTableBasicInfoDto);
     }
     /**
      * Query the columns of the metatable.
      *
      * @param getMetaTableColumnDto
      * @return {@link GetMetaTableColumnResponse.Data}
      */
     @GetMapping("/getTableColumn")
     public GetMetaTableColumnResponse.Data getMetaTableColumn(GetMetaTableColumnDto getMetaTableColumnDto) {
         return metaService.getMetaTableColumn(getMetaTableColumnDto);
     }
     /**
      * Query the partitions of the metatable.
      *
      * @param getMetaTablePartitionDto
      * @return
      */
     @GetMapping("/getTablePartition")
     public GetMetaTablePartitionResponse.Data getMetaTablePartition(GetMetaTablePartitionDto getMetaTablePartitionDto) {
         return metaService.getMetaTablePartition(getMetaTablePartitionDto);
     }
   }

3. Provide a simple interactive interface that allows users to query the details of the metatable, including the basic information, table fields, and table partitions.

   import React from 'react';
   import moment from 'moment';
   import { Form, Grid, Table, Pagination } from '@alifd/next';
   import cn from 'classnames';
   import * as services from '../services';
   import type { TableItem, TableDetailOutput, TableColumn, TableEntity } from '../services/meta';
   
   export interface Props {
     item: TableItem;
   }
   
   const formItemLayout = {
     labelCol: {
       fixedSpan: 6
     },
     wrapperCol: {
       span: 18
     },
     labelTextAlign: 'left' as const,
     colon: true,
   };
   const { Row, Col } = Grid;
   const { Item } = Form;
   const { Column } = Table;
   const DetailContent: React.FunctionComponent<Props> = (props) => {
     const [detail, setDetail] = React.useState<Partial<TableDetailOutput>>({});
     const [columns, setColumns] = React.useState<Partial<TableColumn[]>>([]);
     const [partitions, setPartitions] = React.useState<Partial<TableEntity[]>>([]);
     const [columnsTotal, setColumnsTotal] = React.useState<number>(0);
     const [partitionTotal, setPartitionTotal] = React.useState<number>(0);
     // Implement a method that is used to call the API operation to query the basic information about the metatable.
     const getTableDetail = React.useCallback(async () => {
       const response = await services.meta.getTableDetail({ tableGuid: props.item.tableGuid });
       setDetail(response);
     }, [props.item.tableGuid]);
     // Implement a method that is used to call the API operation to query the fields of the metatable.
     const getTableColumns = React.useCallback(async (pageNum: number = 1) => {
       const response = await services.meta.getMetaTableColumns({
         pageNum,
         tableGuid: props.item.tableGuid,
       });
       setColumns(response.columnList);
       setColumnsTotal(response.totalCount);
     }, [props.item.tableGuid]);
     // Implement a method that is used to call the API operation to query the partitions of the metatable.
     const getTablePartition = React.useCallback(async (pageNumber: number = 1) => {
       const response = await services.meta.getTablePartition({
         pageNumber,
         tableGuid: props.item.tableGuid,
       });
       setPartitions(response.dataEntityList);
       setPartitionTotal(response.totalCount);
     }, []);
     // The permission mappings of the metatable.
     const isVisible = React.useMemo(() => {
       switch (detail.isVisible) {
         case 0:
           return 'Visible to members in the workspace';
         case 1:
           return 'Visible to users within the tenant';
         case 2:
           return 'Visible to all tenants';
         case 3:
           return 'Visible only to the metatable owner';
         default:
           return '';
       }
     }, [detail.isVisible]);
     React.useEffect(() => {
       if (props.item.tableGuid) {
         getTableDetail();
         getTableColumns();
         getTablePartition();
       }
     }, [props.item.tableGuid]);
     // Render the frontend page.
     return (
       <div>
         <Form labelTextAlign="left">
           <Row>
             <Col>
               <Item {...formItemLayout} label="Table GUID">
                 <span>{detail.tableGuid}</span>
               </Item>
             </Col>
             <Col>
               <Item {...formItemLayout} label="Table Name">
                 <span>{detail.tableName}</span>
               </Item>
             </Col>
           </Row>
           <Row>
             <Col>
               <Item {...formItemLayout} label="Workspace">
                 <span className={cn(classes.formContentWrapper)}>{detail.projectName}</span>
               </Item>
             </Col>
             <Col>
               <Item {...formItemLayout} label="Table Owner ID">
                 <span className={cn(classes.formContentWrapper)}>{detail.ownerId}</span>
               </Item>
             </Col>
           </Row>
           <Row>
             <Col>
               <Item {...formItemLayout} label="Table Lifecycle">
                 <span className={cn(classes.formContentWrapper)}>{detail.lifeCycle ? `${detail.lifeCycle} days` : 'Permanent'}</span>
               </Item>
             </Col>
             <Col>
               <Item {...formItemLayout} label="Comment">
                 <span className={cn(classes.formContentWrapper)}>{detail.comment}</span>
               </Item>
             </Col>
           </Row>
           <Row>
             <Col>
               <Item {...formItemLayout} label="Storage Space">
                 <span className={cn(classes.formContentWrapper)}>{`${detail.dataSize} Bytes`}</span>
               </Item>
             </Col>
             <Col>
               <Item {...formItemLayout} label="Partitioned Table or Not">
                 <span className={cn(classes.formContentWrapper)}>{detail.isView ? 'Yes' : 'No'}</span>
               </Item>
             </Col>
           </Row>
           <Row>
             <Col>
               <Item {...formItemLayout} label="Table Visibility">
                 <span className={cn(classes.formContentWrapper)}>{isVisible}</span>
               </Item>
             </Col>
             <Col>
               <Item {...formItemLayout} label="Table Read Count">
                 <span className={cn(classes.formContentWrapper)}>{detail.readCount}</span>
               </Item>
             </Col>
           </Row>
           <Row>
             <Col>
               <Item {...formItemLayout} label="Created At">
                 <span className={cn(classes.formContentWrapper)}>{moment(detail.createTime).format('YYYY-MM-DD HH:mm:ss')}</span>
               </Item>
             </Col>
             <Col>
               <Item {...formItemLayout} label="Last Modified At">
                 <span className={cn(classes.formContentWrapper)}>{moment(detail.lastModifyTime).format('YYYY-MM-DD HH:mm:ss')}</span>
               </Item>
             </Col>
           </Row>
           <Item label="Field Details" colon>
             <Table dataSource={columns}>
               <Column title="Name" dataIndex="columnName" />
               <Column title="Type" dataIndex="columnType" />
               <Column title="Primary Key or Not" dataIndex="isPrimaryKey" />
               <Column title="Foreign Key or Not" dataIndex="isForeignKey" />
               <Column title="Partition Field or Not" dataIndex="isPartitionColumn" />
               <Column title="Comment" dataIndex="comment" />
             </Table>
             <Pagination
               total={columnsTotal}
               onChange={getTableColumns}
               showJump={false}
               className={cn(classes.tablePaginationWrapper)}
             />
           </Item>
           <Item label="Partition Details" style={{ marginTop: 32, marginBottom: 32 }} colon>
            <Table
             dataSource={partitions}
             emptyContent={<span>Non-partitioned Table</span>}
           >
               <Column title="Name" dataIndex="partitionName" />
               <Column title="Type" dataIndex="partitionType" />
               <Column title="Partition Size" dataIndex="dataSize" cell={value => `${value} bytes`} />
               <Column title="Comment" dataIndex="comment" />
             </Table>
             <Pagination
               total={partitionTotal}
               onChange={getTablePartition}
               showJump={false}
               className={cn(classes.tablePaginationWrapper)}
             />
           </Item>
         </Form>
       </div>
     );
   }
   
   export default DetailContent;
                           

4. After you write the preceding code, you can deploy and run the project code offline. For information about how to deploy and run the project code, see Common operations: Deploy and run the code offline.

Practice 3: Query the ancestor and descendant lineages of a metatable

This section describes how to call the ListLineage operation to query the ancestor and descendant lineages of a metatable.

1. Write the getMetaTableLineage method in meta-service-proxy to call the getMetaTableLineage operation in the OpenAPI module and process input parameters.

   package com.aliyun.dataworks.services;
   
   import com.aliyun.dataworks.dto.*;
   import com.aliyuncs.IAcsClient;
   import com.aliyuncs.dataworks_public.model.v20200518.*;
   import com.aliyuncs.exceptions.ClientException;
   import com.aliyuncs.exceptions.ServerException;
   import com.aliyuncs.utils.StringUtils;
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.stereotype.Service;
   import org.springframework.util.CollectionUtils;
   
   /**
    * @author dataworks demo
    */
   @Service
   public class MetaServiceProxy {
     @Autowired
     private DataWorksOpenApiClient dataWorksOpenApiClient;
     /**
      * DataWorks OpenAPI : GetMetaTableLineage
      * Link : https://www.alibabacloud.com/help/doc-detail/173927.html
      *
      * @param getMetaTableLineageDto
      */
     public GetMetaTableLineageResponse.Data getMetaTableLineage(GetMetaTableLineageDto getMetaTableLineageDto) {
       try {
         IAcsClient client = dataWorksOpenApiClient.createClient();
         GetMetaTableLineageRequest getMetaTableLineageRequest = new GetMetaTableLineageRequest();
         // The GUID of the metatable. 
         getMetaTableLineageRequest.setTableGuid(getMetaTableLineageDto.getTableGuid());
         // The drilling direction for the field. The value up indicates the ancestor lineage, and the value down indicates the descendant lineage. 
         getMetaTableLineageRequest.setDirection(getMetaTableLineageDto.getDirection());
         // The paging information. 
         getMetaTableLineageRequest.setNextPrimaryKey(getMetaTableLineageDto.getTableGuid());
         // The number of entries to return on each page. Default value: 10. Maximum value: 100. 
         getMetaTableLineageRequest.setPageSize(getMetaTableLineageDto.getPageSize());
         // The ID of the EMR cluster. This parameter is required only if you query data in an EMR compute engine instance. 
         getMetaTableLineageRequest.setClusterId(getMetaTableLineageDto.getClusterId());
         // The name of the database. 
         getMetaTableLineageRequest.setDatabaseName(getMetaTableLineageDto.getDatabaseName());
         // The name of the metatable.
         getMetaTableLineageRequest.setTableName(getMetaTableLineageDto.getTableName());
         // The type of the data source. Valid values: odps and emr. 
         getMetaTableLineageRequest.setDataSourceType(getMetaTableLineageDto.getDataSourceType());
         GetMetaTableLineageResponse acsResponse = client.getAcsResponse(getMetaTableLineageRequest);
         for (GetMetaTableLineageResponse.Data.DataEntityListItem dataEntityListItem : acsResponse.getData()
           .getDataEntityList()) {
           // The name of the table. 
           System.out.println(dataEntityListItem.getTableName());
           // The GUID of the metatable. 
           System.out.println(dataEntityListItem.getTableGuid());
           // The time when the metatable was created. 
           System.out.println(dataEntityListItem.getCreateTimestamp());
         }
         return acsResponse.getData();
       } catch (ServerException e) {
         e.printStackTrace();
       } catch (ClientException e) {
         e.printStackTrace();
         // The ID of the request.
         System.out.println(e.getRequestId());
         // The error code.
         System.out.println(e.getErrCode());
         // The error message.
         System.out.println(e.getErrMsg());
       }
       return null;
     }
   }

2. Define the getMetaTableLineage method as an entrance that allows access from frontend clients.

   package com.aliyun.dataworks.demo;
   
   import com.aliyun.dataworks.dto.*;
   import com.aliyun.dataworks.services.MetaServiceProxy;
   import com.aliyuncs.dataworks_public.model.v20200518.*;
   import org.springframework.beans.factory.annotation.Autowired;
   import org.springframework.web.bind.annotation.GetMapping;
   import org.springframework.web.bind.annotation.PostMapping;
   import org.springframework.web.bind.annotation.RequestMapping;
   import org.springframework.web.bind.annotation.RestController;
   import org.springframework.web.bind.annotation.CrossOrigin;
   
   
   /**
    * Demonstrate how to build a custom metadata platform by using the OpenAPI module.
    *
    * @author dataworks demo
    */
   @RestController
   @RequestMapping("/meta")
   public class MetaRestController {
     @Autowired
     private MetaServiceProxy metaService;
     /**
      * Query the data lineage of the metatable.
      *
      * @param getMetaTableLineageDto
      * @return
      */
     @GetMapping("/getTableLineage")
     public GetMetaTableLineageResponse.Data getMetaTableLineage(GetMetaTableLineageDto getMetaTableLineageDto) {
         return metaService.getMetaTableLineage(getMetaTableLineageDto);
     }
   }

3. Provide an interface that allows users to query the data lineage of a metatable by calling API operations. Then, render the interface.

   import React from 'react';
   import Graphin, { Behaviors } from '@antv/graphin';
   import type { IUserNode, IUserEdge, GraphinData, GraphEvent } from '@antv/graphin';
   import * as services from '../services';
   import type { TableItem, GetTableLineageOutput, LineageEntity } from '../services/meta';
   import '@antv/graphin/dist/index.css';
   
   export interface Props {
     item: TableItem;
   }
   
   function transNode(entity: LineageEntity | TableItem, direction?: string): IUserNode {
     return {
       id: entity.tableGuid,
       label: entity.tableName,
       data: {
         ...entity,
         direction,
       },
       style: {
         label: {
           value: entity.tableName,
         }
       },
     }
   }
   function transEdge(source: LineageEntity | TableItem, target: LineageEntity | TableItem): IUserEdge {
     return {
       source: source.tableGuid,
       target: target.tableGuid,
     };
   }
   function parse(
     source: LineageEntity | TableItem,
     data: GetTableLineageOutput,
     direction: string,
   ): [IUserNode[], IUserEdge[]] {
     const nodes: IUserNode[] = [];
     const edges: IUserEdge[] = [];
     data.dataEntityList.forEach((entity) => {
       nodes.push(transNode(entity, direction));
       if (direction === 'down') {
         edges.push(transEdge(source, entity));
       } else {
         edges.push(transEdge(entity, source));
       }
     });
     return [nodes, edges];
   }
   function mergeNodes(prev: IUserNode[], next: IUserNode[]) {
     const result: IUserNode[] = prev.slice();
     next.forEach((item) => {
       const hasValue = prev.findIndex(i => i.id === item.id) >= 0;
       !hasValue && result.push(item);
     });
     return result;
   }
   function mergeEdges(source: IUserEdge[], target: IUserEdge[]) {
     const result: IUserEdge[] = source.slice();
     target.forEach((item) => {
       const hasValue = source.findIndex(i => i.target === item.target && i.source === item.source) >= 0;
       !hasValue && result.push(item);
     });
     return result;
   }
   
   const { ActivateRelations, DragNode, ZoomCanvas } = Behaviors;
   const LineageContent: React.FunctionComponent<Props> = (props) => {
     const ref = React.useRef<Graphin>();
     const [data, setData] = React.useState<GraphinData>({ nodes: [], edges: [] });
     // Call the backend API operation to obtain the lineage information.
     const getTableLineage = async (
       collection: GraphinData,
       target: TableItem | LineageEntity,
       direction: string,
     ) => {
       if (!direction) {
         return collection;
       }
       const response = await services.meta.getTableLineage({
         direction,
         tableGuid: target.tableGuid,
       });
       const [nodes, edges] = parse(target, response, direction);
       collection.nodes = mergeNodes(collection.nodes!, nodes);
       collection.edges = mergeEdges(collection.edges!, edges);
       return collection;
     };
     // Obtain data and render the page during initialization.
     const init = async () => {
       let nextData = Object.assign({}, data, { nodes: [transNode(props.item)] });
       nextData = await getTableLineage(nextData, props.item, 'up');
       nextData = await getTableLineage(nextData, props.item, 'down');
       setData(nextData);
       ref.current!.graph.fitCenter();
     };
     React.useEffect(() => {
       ref.current?.graph && init();
     }, [
       ref.current?.graph,
     ]);
     // Process the event that is triggered when you click the metatable and initiate a request to obtain data.
     React.useEffect(() => {
       const graph = ref.current?.graph;
       const event = async (event: GraphEvent) => {
         const source = event.item?.get('model').data;
         let nextData = Object.assign({}, data);
         nextData = await getTableLineage(nextData, source, source.direction);
         setData(nextData);
       };
       graph?.on('node:click', event);
       return () => {
         graph?.off('node:click', event);
       };
     }, [
       data,
       ref.current?.graph,
     ]);
     // Render the lineage graph.
     return (
       <div>
         <Graphin
           data={data}
           ref={ref as React.LegacyRef<Graphin>}
           layout={{
             type: 'dagre',
             rankdir: 'LR',
             align: 'DL',
             nodesep: 10,
             ranksep: 40,
           }}
         >
           <ActivateRelations />
           <DragNode disabled />
           <ZoomCanvas enableOptimize />
         </Graphin>
       </div>
     );
   };
   
   export default LineageContent;
                           

4. After you write the preceding code, you can deploy and run the project code offline. For information about how to deploy and run the project code, see Common operations: Deploy and run the code offline.

Practice 4: Find the node corresponding to the table

When the caliber of a table changes, you can use the DataWorks API, OpenData, and message subscription methods to analyze the lineage of downstream tasks and find the node corresponding to the table. Perform the following steps:

1. View the lineage of a table by ListLineage

2. Find the list of affected tables based on the field lineage or table lineage, and obtain the task ID of the table by GetMetaTableOutput based on the table list.

3. Obtain the task details by GetNode based on the specified task ID to confirm the impact on your business.

Common operations: Deploy and run the code offline

1. Prepare the environment.

   Make sure that the following dependencies are prepared: Java 8 or later, Maven, runtime environment, and pnpm. Maven is a build automation tool for Java. You can run the following commands to check whether the preceding preparations are complete:

   npm -v // If the installation is successful, the version is displayed in the command output. If the installation fails, an error that indicates no command is available is reported.
   java -version // If the installation is successful, the version is displayed in the command output. If the installation fails, an error that indicates no command is available is reported.
   pnpm -v // If the installation is successful, the version is displayed in the command output. If the installation fails, an error that indicates no command is available is reported.

2. Download the project code and run the following command:

   Download link for the project file: meta-api-demo.zip.

   pnpm i

3. In the sample project, find the application.properties file in the backend/src/main/resources directory and modify the core parameters in the file.

   • Set the api.access-key-id and api.access-key-secret parameters to the AccessKey ID and AccessKey Secret of your Alibaba Cloud account.

     Note

     You can obtain the information about your Alibaba Cloud account on the Security Management page in the Alibaba Cloud Management Console.

   • Configure the api.region-id and api.endpoint parameters based on the information about the region of the API operations that you want to call.

   Configure other parameters based on your business requirements. The following figure shows sample configurations.

   

4. Run the following command in the project root directory to run the sample code:

   npm run dev