DataWorks:Metadata collection

MaxCompute default crawler

This crawler collects metadata from MaxCompute projects in your account. On the details page, use Modify Data Scope to select projects and Permission Configurations to set metadata visibility within the tenant.

1. In the Built-in section of the metadata collection page, find the MaxCompute Default Crawler card and click Details.

2. The MaxCompute Default Crawler details page contains the Basic Information and Data Scope tabs.

   • Basic Information: Displays basic attributes of the crawler, such as collection type and mode. This information is read-only.

   • Data Scope: Manages which MaxCompute projects to collect.

3. Modify collection scope:

   1. Click Data Scope and click Modify Data Scope.

   2. In the dialog box, select or clear the MaxCompute projects to collect.

      Important

      The default scope includes all MaxCompute projects bound to workspaces in the current region under the current tenant. After the scope is modified, only metadata objects within the scope are visible in Data Map. Metadata that is not selected will be invisible.

   3. Click OK to save the changes.

4. Configure metadata visibility:

   • In the Data Scope list, find the target project and click Actions in the Permission Configurations column.

   • Select a visibility policy based on your data governance requirements:

     • Public Within Tenant: All members in the tenant can search for and view metadata of this project.

     • Only members in the associated workspace can search and view.: Only members of specific workspaces can access metadata of this project, ensuring data isolation.

DLF Default Crawler

The DLF Default Crawler collects metadata from Data Lake Formation (DLF) within your account.

1. In the Built-in section of the metadata collection page, find the DLF Default Crawler card and click Details to view basic information.

2. Click the Data Scope tab to view the list of DLF Catalogs included in the collection scope and their table counts.

   By default, all accessible Catalogs (including DLF and DLF-Legacy versions) are collected.

Create custom crawler

1. In the custom crawler list section of the metadata collection page, click Create Metadata Collection.

2. Select collection type: On the type selection page, select the target data source type to collect, such as Hologres or StarRocks.

3. Configure basic information and resource group:

   • Basic Configurations:

     • Select Workspace: Select the workspace containing the data source.

     • Select Data Source: Select a created target data source from the drop-down list. After selection, the system automatically displays details of the data source.

     • Name: Enter a name for the crawler for future identification. The default name is the same as the data source name.

   • Resource Group Configuration:

     • Resource Group: Select a resource group to run the collection task.

     • Test Network Connectivity: This step is critical. Click Test Network Connectivity to ensure the resource group can successfully access the data source.

       Important

       • Check the data source for whitelist restrictions. If you need to collect metadata with whitelist access control enabled, see Overview of network connectivity solutions and Configure a whitelist to configure whitelist permissions.

       • If the data source does not have whitelist restrictions, see Network connectivity and operations on resource groups for network connectivity configuration.

       • If the connectivity test fails with error: backend service call failed: test connectivity failed.not support data type, contact technical support to upgrade the resource group.

4. Configure metadata collection:

   • Collection Scope: Define the databases (Database/Schema) to collect. If the data source is database-granular, the corresponding database is selected by default. You can select additional databases outside the data source.

     Important

     • A database can be configured in only one crawler. If a database cannot be selected, it is already being collected by another crawler.

     • If you narrow the collection scope, metadata outside the scope becomes unsearchable in Data Map.

5. Configure Intelligent Enhancement Settings and Collection Plan:

   • Intelligent Enhancement Settings (Beta):

     • AI Collection Description: When enabled, the system uses LLMs to automatically generate business descriptions for your tables and fields after metadata collection, greatly improving metadata readability and usability. After collection is complete, you can view the AI-generated information (such as table remarks and field descriptions) on the details page of the table object in Data Map.

   • Collection Plan:

     • Trigger Mode: Select Manual or Periodic.

       • Manual: The crawler runs only when manually triggered. This applies to one-time or on-demand collection.

       • Periodic: Configure a scheduled task (such as monthly, daily, weekly, or hourly). The system will automatically update metadata periodically.

         To configure a minute-level scheduled task, select hourly collection and check all minute options to achieve a 5-minute interval task.

         Important

         Periodic collection is supported only for production environment data sources.

6. Save configuration: Click Save or Save and Run to complete the creation of the crawler.

Manage custom crawlers

After a crawler is created, it appears in the custom list. You can perform the following management operations:

• List operations: In the list, you can directly Run, Stop, or Delete the crawler. Use the Filter and Search features at the top to quickly locate the target crawler.

  Important

  Deleting a metadata crawler removes its collected metadata objects from Data Map. Users cannot search for or view these objects. Caution: This action cannot be undone.

• View details and logs: Click the crawler name to view its details.

  • Basic Information: View all configuration items of the crawler.

  • Data Scope: View or Modify Data Scope.

    If viewed before collection, the table count and latest update time will be empty.

    The following data sources do not support scope modification: EMR Hive, CDH Hive, Lindorm, ElasticSearch, Tablestore (OTS), MongoDB, and AnalyticDB for Spark within AnalyticDB for MySQL.

  • Run Logs: Track the execution history of each collection task. You can view the start time, duration, status, and volume of collected data. When a task fails, clicking View Logs is the key entry point for locating and resolving issues.

• Manually execute collection: In the upper-right corner, click Collect Metadata to immediately trigger a collection task. Use this to immediately view a newly created table in Data Map.

Data tables

Task code

Data Map supports code search and quick location. The following table describes the supported scope.

API assets

Data Map supports viewing DataService Studio API metadata:

AI assets

Data Map supports viewing and managing AI assets, and provides AI asset lineage to trace the origin, usage, and evolution of data and models. The following table describes the support for AI assets.

Workspace

Data Map supports viewing workspace metadata: