MaxCompute:Python 2 UDTF

You can use MaxCompute Studio to write UDTF code in Python 2. The UDTF code must contain the following information:

• Encoding declaration: optional.

  The declaration format is #coding:utf-8 or # -*- coding: utf-8 -*-. The two formats are equivalent. If Chinese characters appear in UDTF code that is written in Python 2, an error is returned when you run the UDTF. To resolve this issue, you must add an encoding declaration to the header of the code.

• Module import: required.

  UDTF code must include from odps.udf import annotate and from odps.udf import BaseUDTF. from odps.udf import annotate is used to import the function signature module. This way, MaxCompute can identify the function signature that is defined in the code. from odps.udf import BaseUDTF is the base class for Python UDTFs. You must use this class to implement methods such as process and close in derived classes.

  If you want to reference files or tables in UDTF code, UDTF code must include from odps.distcache import get_cache_file or from odps.distcache import get_cache_table.

• Function signature: optional.

  The function signature is in the @annotate(<signature>) format. The signature parameter is used to define the data types of the input parameters and return value of a UDTF. If you do not specify a function signature, input parameters of any data type can be matched when you call a UDTF in SQL statements. As a result, the data types of the return values cannot be inferred and all output parameters are of the STRING type. For more information about function signatures, see Function signatures and data types.

• Custom Python class (derived class): required.

  A custom Python class is the organizational unit of UDTF code. This class defines the variables and methods that are used to meet your business requirements. In UDTF code, you can reference third-party libraries that are built in MaxCompute or reference files or tables. For more information, see Third-party libraries or Resource reference.

• Methods to implement Python classes: required.

  Four methods can be used to implement Python classes. The following table describes these methods.

  Method	Description
  BaseUDTF.init()	The initialization method. To implement this method for a derived class, you must call the super(BaseUDTF, self).init() initialization method of the base class when you start to run code. The INIT method is called only once throughout the lifecycle of a UDTF. This method is called only before the first record is processed. If a UDTF needs to save internal states, all states can be initialized by using this method.
  BaseUDTF.process([args, ...])	The process function is called once for each SQL record. The parameters of the process function are the input parameters of the UDTF that is specified in SQL statements.
  BaseUDTF.forward([args, ...])	The output method of a UDTF. This method is called by user code. One output record is generated each time the forward method is called. The parameters in the forward method are the UDTF output parameters that are specified in SQL statements. If no function signature is specified in the Python code, all output values must be converted into the STRING type when the forward method is called.
  BaseUDTF.close()	The method to terminate a UDTF. This method is called only once. It is called only before the last record is processed.

The following example shows the UDTF code.

#coding:utf-8
# Import the function signature module and the base class. 
from odps.udf import annotate
from odps.udf import BaseUDTF
# The function signature. 
@annotate('string -> string')
# The custom Python class. 
class Explode(BaseUDTF):
# Methods used to implement Python classes. 
   def process(self, arg):
       props = arg.split(',')
       for p in props:
           self.forward(p)

MaxCompute allows you to write Python 2 UDTFs in Python 2.7 and run the UDTF code in a sandbox environment. In this environment, the following operations are prohibited:

• Read data from and write data to local files.
• Start subprocesses.
• Start threads.
• Enable socket communication.
• Use other systems to call Python 2 UDFs.

Due to these limits, the code that you upload must be written by using Python standard libraries. If modules or C extension modules in Python standard libraries are involved in the preceding operations, these modules cannot be used. Take note of the following points about modules in Python standard libraries:

• All the modules that are implemented based on Python standard libraries and do not depend on extension modules are available.
• The following C extension modules are available:
  • array and audioop
  • binascii and bisect
  • cmath, _codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw, _collections, and cStringIO
  • datetime
  • _functools and future_builtins
  • _heapq and _hashlib
  • itertools
  • _json
  • _locale and _lsprof
  • math, _md5, and _multibytecodec
  • operator
  • _random
  • _sha256, _sha512, _sha, _struct, and strop
  • time
  • unicodedata
  • _weakref
  • cPickle
• When you run UDF code in a sandbox environment, the maximum size of data that can be written to the standard output (sys.stdout) or standard error output (sys.stderr) is 20 KB. If the size exceeds 20 KB, extra characters are ignored.

Third-party libraries, such as NumPy, are installed in the Python 2 environment of MaxCompute as supplements to standard libraries.

Format of function signature:

@annotate(<signature>)

signature is a function signature string. This parameter is used to identify the data types of the input parameters and return values. When a UDTF is run, the input parameters and return values of the UDTF must be of the same data type as those specified in the function signature. The system checks whether the UDTF complies with the definition of the function signature during semantics parsing. If the data types of the UDTF are inconsistent with the data types specified in the function signature, an error is returned. The signature is in the following format:

'arg_type_list -> type_list'

Parameter description:

• type_list: indicates the data types of return values. A UDTF can return multiple columns. The following data types are supported: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), complex data types (ARRAY, MAP, and STRUCT), and nested complex data types.
• arg_type_list: indicates the data types of input parameters. If multiple input parameters are used, specify multiple data types and separate them with commas (,). The following data types are supported: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), CHAR, VARCHAR, complex data types (ARRAY, MAP, and STRUCT), and nested complex data types.
  arg_type_list can also be set to an asterisk (*) or left empty.

  • If arg_type_list is set to an asterisk (*), a random number of input parameters are used.
  • If arg_type_list is left empty, no input parameters are used.

The following table provides examples of valid function signatures.

Function signature	Description
@annotate('bigint,boolean->string,datetime')	The data types of the input parameters are BIGINT and BOOLEAN. The data types of the return values are STRING and DATETIME.
@annotate('*->string, datetime')	A random number of input parameters are used and the data types of the return values are STRING and DATETIME.
@annotate('->double, bigint, string')	No input parameters are used, and the data types of the return values are DOUBLE, BIGINT, and STRING.
@annotate("array<string>,struct<a1:bigint,b1:string>,string->map<string,bigint>,struct<b1:bigint>")	The data types of the input parameters are ARRAY, STRUCT, and MAP. The data types of the return values are MAP and STRUCT.

The following sample code shows how to reference files and tables.

# -*- coding: utf-8 -*-
from odps.udf import annotate
from odps.udf import BaseUDTF
from odps.distcache import get_cache_file
from odps.distcache import get_cache_table
@annotate('string -> string, bigint')
class UDTFExample(BaseUDTF):
    """Read pageid and adid in the file and table to generate dict.
    """
    def __init__(self):
        import json
        cache_file = get_cache_file('test_json.txt')
        self.my_dict = json.load(cache_file)
        cache_file.close()
        records = list(get_cache_table('table_resource1'))
        for record in records:
            self.my_dict[record[0]] = [record[1]]
    """Enter pageid and generate pageid and all adid values.
    """
    def process(self, pageid):
        for adid in self.my_dict[pageid]:
            self.forward(pageid, adid)

After you develop a Python 2 UDTF by following the instructions in Development process, you can use MaxCompute SQL to call the Python 2 UDTF. The following steps describe how to call a Python 2 UDTF:

• Use a UDF in a MaxCompute project: The method is similar to that of using built-in functions.
• Use a UDF across projects: Use a UDF of Project B in Project A. The following statement shows an example: select B:udf_in_other_project(arg0, arg1) as res from table_t;. For more information about resource sharing across projects, see Cross-project resource access based on packages.