DataWorks:Export tasks from open source engines

Limits

For Airflow tasks, you can export only tasks of Airflow 1.10.x. In addition, the export of Airflow tasks depends on Python 3.6 or later.

Export a task from Oozie

Export requirements

The package must contain XML-formatted definition files and configuration files of a task. The package is exported in the ZIP format.

Structure of the package

Export a job from Azkaban

Download a flow

You can download a specific flow in the Azkaban console.

1. Log on to the Azkaban console and go to the Projects page.

2. Select a project whose package you want to download. On the project page, click Flows to show all flows of the project.

3. Click Download in the upper-right corner of the page to download the package of the project.

   Native Azkaban packages can be exported. No format limit is imposed on the packages of Azkaban. The exported package in the ZIP format contains information about all jobs and relationships of a specific project of Azkaban. You can directly upload the ZIP package exported from the Azkaban console to the Scheduling Engine Import page in DataWorks.

Conversion logic

The following table describes the mappings between Azkaban items and DataWorks items and the conversion logic.

Export a task from Airflow

Procedure

1. Go to the runtime environment of Airflow.

2. Use the Python library of Airflow to load the directed acyclic graph (DAG) folder that is scheduled on Airflow. The DAG Python file is stored in the DAG folder.

3. Use the export tool to read the task information and dependencies stored in the DAG Python file based on the Python library of Airflow in memory. Then, write the generated DAG information to a JSON file and export the file.

   You can download the export tool on the Scheduling Engine Export page of Cloud tasks in DataWorks Migrant Assistant. For information about how to go to the Scheduling Engine Export page, see Export a task of another open source engine.

Usage notes for the export tool

Usage notes for the export tool:

1. Run the following command to decompress the airflow-exporter.tgz package:

   tar zxvf airflow-exporter.tgz

2. Run the following command to set the PYTHONPATH parameter to the directory of the Python library:

   export PYTHONPATH=/usr/local/lib/python3.6/site-packages

3. Run the following command to export the task from Airflow:

   cd airflow-exporter
   python3.6 ./parser -d /path/to/airflow/dag/floder/ -o output.json

4. Run the following command to compress the exported output.json file into a ZIP package:

   zip out.zip output.json

After the ZIP package is generated, you can perform the following operations to create an import task to import the ZIP package to DataWorks: Go to the Migrant Assistant page in the DataWorks console. In the left-side navigation pane, choose Cloud tasks > Scheduling Engine Import. For more information, see Import tasks of open source engines.

Export a node from DolphinScheduler

How it works

The DataWorks export tool obtains the JSON configurations of a process in DolphinScheduler by calling the API operation that is used to export multiple processes from DolphinScheduler at a time. The export tool generates a ZIP file based on the JSON configurations. In the left-side navigation pane of the Migrant Assistant page, you can choose Cloud tasks > Scheduling Engine Import, and create an import task of the DolphinScheduler type to import the ZIP file. DataWorks Migration Assistant parses and converts code and dependencies of nodes in a process in the ZIP file into valid file configurations for related DataWorks nodes. For information about how to import a task of an open source scheduling engine, see Scheduling Engine Import.

Limits

• Limits on version: DolphinScheduler 1.3.x, 2.x, 3.x allows you to export DolphinScheduler nodes.

• Limits on node type conversion:

  • SQL nodes: Only some types of compute engines support the conversion of SQL nodes. During the node type conversion, the syntax of SQL code is not converted and the SQL code is not modified.

  • Cron expressions: In specific scenarios, cron expressions may be pruned or cron expressions may not be supported. You must check whether the scheduling time that is configured meets your business requirements. For information about scheduling time, see Configure time properties.

  • Python nodes: DataWorks does not provide Python nodes. DataWorks can convert a Python node in DolphinScheduler into a Python file resource and a Shell node that references the Python file resource. However, issues may occur when scheduling parameters of the Python node are passed. Therefore, debugging and checks are required. For information about scheduling parameters, see Configure and use scheduling parameters.

  • Depend nodes: DataWorks cannot convert Depend nodes for which cross-cycle scheduling dependencies are configured. If cross-cycle scheduling dependencies are configured for a Depend node, DataWorks converts the dependencies into the same-cycle scheduling dependencies of the mapped auto triggered node in DataWorks. For information about how to configure same-cycle scheduling dependencies, see Configure same-cycle scheduling dependencies.

Conversion logic

The following table describes the mappings between DolphinScheduler items and DataWorks items and the conversion logic.

{current_dataworks_project_name}.{dolphin_project_name}.{dolphin_process_name}.{dolphin_task_name}

Environment preparations

• Download the export tool package.

• Dependency settings: Install Java Development Kit (JDK) 1.8 or later, and Python 2.7 or later.

Export procedure

1. Decompress the export tool package.

   Run the following commands to decompress the export tool package:

   $ tar xzvf migrationx-reader.zip
   $ cd migrationx-reader/

2. Generate a token used to call DolphinScheduler APIs.

   For information about how to generate a token, see DolphinScheduler documentation.

3. Export a ZIP file.

   Run the following command to export the required ZIP file:

   $ python ./bin/reader.py -a dolphinscheduler -e http://dolphinschedulerhost:port -t token -v 1.3.9 -p project_name -f ds_dump.zip

After you export the ZIP file, choose Cloud tasks > Scheduling Engine Import in the left-side navigation pane of the Migrant Assistant page, and create an import task of the DolphinScheduler type to import the ZIP file. DataWorks Migration Assistant parses and converts code and dependencies of nodes in a process in the ZIP file into valid file configurations for related DataWorks nodes. For information about how to import a task of an open source scheduling engine, see Scheduling Engine Import.

Export tasks from other open source engines

DataWorks provides a standard template for you to export tasks of open source engines other than Oozie, Azkaban, Airflow, and DolphinScheduler. Before you export a task, you must download the standard template and modify the content based on the file structure in the template. You can go to the Scheduling Engine Export page to download the standard template and view the file structure.

1. Log on to the DataWorks console. In the top navigation bar, select the desired region. Then, choose Data Modeling and Development > DataStudio in the left-side navigation pane. On the page that appears, select the desired workspace from the drop-down list and click Go to DataStudio.

2. On the DataStudio page, click the icon in the upper-left corner and choose All Products > More > Migration Assistant.

3. In the left-side navigation pane, choose Cloud tasks > Scheduling Engine Export.

4. Click the Standard Template tab.

5. On the Standard Template tab, click standard format Template to download the template.

6. Modify the content of the template to generate a package that you want to export.