DataWorks:Use a PyODPS node to reference a third-party package

Limits

• Due to the specifications of resources in the resource group that is used to run a node, we recommend that you use a PyODPS node to locally process no more than 50 MB of data. If a PyODPS node processes more than 50 MB of data, an out-of-memory (OOM) exception may occur, and the system may report Got killed. We recommend that you do not write excessive data processing code for a PyODPS node. For more information, see Overview.

• If the system reports Got killed, the memory usage exceeds the limit, and the system terminates the related processes. We recommend that you do not perform local data operations. However, the limits on the memory usage do not apply to SQL or DataFrame tasks that are initiated by PyODPS. Take note that to_pandas tasks are excluded.

• You can use the NumPy and pandas libraries that are pre-installed in DataWorks to run functions other than UDFs on an exclusive resource group for scheduling. Third-party packages that contain binary code are not supported.

• For compatibility reasons, options.tunnel.use_instance_tunnel is set to False in DataWorks by default. If you want to globally enable InstanceTunnel, you must set this parameter to True.

Reference a common Python script

1. Go to the DataStudio page.

   Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose Data Development and Governance > Data Development. On the page that appears, select the desired workspace from the drop-down list and click Go to Data Development.

2. Create a Python resource

   1. On the DataStudio page, move the pointer over the icon and choose Create Resource > MaxCompute > Python.

      Alternatively, you can click the name of the desired workflow in the Business Flow section, right-click MaxCompute, and then choose Create Resource > Python.

   2. In the Create Resource dialog box, configure the Name parameter. In this example, the Name parameter is set to pyodps_packagetest.py.

      Important

      The resource name can contain only letters, digits, periods (.), underscores (_), and hyphens (-). It must end with .py.

   3. Click Create.

   4. On the configuration tab of the newly created Python resource, enter the common Python script that you want to reference. In this example, the following script is used:

      # import os
      # print os.getcwd()
      # print os.path.abspath('.')
      # print os.path.abspath('..')
      # print os.path.abspath(os.curdir)
      
      def printname():
          print 'test2'
      print 123

   5. Click the icon in the top toolbar.

3. Create a PyODPS 2 node.

   1. In the Business Flow section, find the workflow in which you want to create a PyODPS 2 node, right-click MaxCompute, and then choose Create Node > PyODPS 2.

   2. In the Create Node dialog box, configure the Name parameter. In this example, the Name parameter is set to pyodps_testpackage.

      Note

      The node name cannot exceed 128 characters in length and can contain letters, digits, underscores (_), and periods (.).

   3. Click Confirm.

4. Open the configuration tab of the newly created PyODPS 2 node. Then, right-click the name of the Python resource in the Resource folder of your workflow and select Insert Resource Path.

   After the resource is referenced, the ##@resource_reference{"pyodps_packagetest.py"} statement is automatically written in the code editor of the PyODPS 2 node.

5. Enter the code that is used to reference the common Python script in the code editor of the PyODPS 2 node. In this example, the following code is used:

   ##@resource_reference{"pyodps_packagetest.py"} # This statement is required to reference the created Python resource. 
   
   import sys
   import os
   sys.path.append(os.path.dirname(os.path.abspath('pyodps_packagetest.py'))) # Import the resource to the workspace. 
   import pyodps_packagetest # Reference the resource. You must delete the .py suffix in the resource name. 
   pyodps_packagetest.printname() # Call the method.

6. Click the icon in the top toolbar and view the results on the Runtime Log tab in the lower part of the configuration tab.

   

Reference a third-party open source package