FAQ about MaxCompute Python UDFs - MaxCompute - Alibaba Cloud Documentation Center

This topic provides answers to some frequently asked questions about MaxCompute user-defined functions (UDFs) that are written in Python.

Classes and resources

The following class or resource issues may occur:

Problem description 1: When I call a MaxCompute UDF, the following error message is reported: function 'xxx' cannot be resolved.
- Possible causes:
  - Cause 1: The project that you use is not the project in which the MaxCompute UDF is registered. For example, if you register a MaxCompute UDF in a development project but call the MaxCompute UDF in a production project, the error is reported.
  - Cause 2: The classes or resources of the MaxCompute UDF are invalid.
  - Cause 3: The type of a resource on which the MaxCompute UDF depends is invalid. For example, you upload a Python file, which is of the PY type. However, get_cache_file in the MaxCompute UDF code requires a resource of the FILE type.
  - Cause 4: The data of a resource on which the MaxCompute UDF depends is not the latest. If you upload a resource to MaxCompute by using DataWorks, a delay may occur. In this case, the data of the resource on which the MaxCompute UDF depends is not the latest.
  - Cause 5: The Python version is not supported. By default, MaxCompute uses Python 2 to run jobs. If non-ASCII characters exist in the Python code, an error is reported when you run the code.
- Solutions:
  - Solution 1: On the MaxCompute client, run the list functions; command in the project where the error is reported. If the MaxCompute UDF is not contained in the command output, the project is not the project in which the MaxCompute UDF is registered. Go to the project in which the MaxCompute UDF is registered and call the MaxCompute UDF again.
  - Solution 2: On the MaxCompute client, run the desc function <function_name>; command and check the values of Class and Resources in the command output.
    If the value of the Class or Resources parameter is invalid, run the create function <function_name> as <'package_to_class'> using <'resource_list'>; command to register the UDF again. Configure the package_to_class parameter in the Python script name.Class name format. Use the resource_list parameter to specify all the resources that you want to reference in MaxCompute, such as files, tables, compressed packages, and third-party packages.
    For more information about how to register a function, see Create a UDF.
  - Solution 3: On the MaxCompute client, run the desc resource <resource_name>; command. Check whether the resource type that is specified by the Type parameter in the command output is valid. If the resource type is invalid, run the add <file_type> <file_name>; command to upload the resource again.
    - If the resource reference method that is used in the MaxCompute UDF code is get_cache_file, a file is referenced and the resource type must be FILE.
    - If the resource reference method that is used in the MaxCompute UDF code is get_cache_table, a table is referenced and the resource type must be TABLE.
    - If the resource reference method that is used in the MaxCompute UDF code is get_cache_archive, a compressed package is referenced and the resource type must be ARCHIVE.
    For more information about how to upload resources, see Add resources.
  - Solution 4: On the MaxCompute client, run the desc resource <resource_name>; command. Check whether the resource data is the latest based on the value of the LastModifiedTime parameter in the command output. You must use the latest resource in the MaxCompute UDF.
  - Solution 5: Add #coding:utf-8 or # -*- coding: utf-8 -*- to the Python code header to declare the encoding format. You can also add the set odps.sql.python.version=cp37; command before the SQL statement that is used to call the MaxCompute UDF and submit the command together with the SQL statement. This way, Python 3 is used to run jobs.
Problem description 2: When I call a MaxCompute UDF in which get_cache_archive('xxx.zip') is configured, one of the following errors is reported: IOError: Download resource: xxx.zip failed, odps.distcache.DistributedCacheError, and fuxi job failed: Download resource failed: xxx.zip.
- Possible causes:
  - Cause 1: The compressed package does not exist. When you register the MaxCompute UDF, you do not specify a compressed package in the command.
  - Cause 2: The resource type of the compressed package is not ARCHIVE.
  - Cause 3: The package name or file name extension in the code is inconsistent with the actual package name or file name extension. For example, xxx.zip is specified in the code but the uploaded file is xxx.tar.gz. In this case, the package is decompressed in the ZIP format. As a result, the decompression fails and an error is reported.
  - Cause 4: Two UDFs that are in the same job depend on resources that have the same name. However, the resources exist in different projects.
- Solutions:
  - Solution 1: On the MaxCompute client, run the desc function <function_name>; command. Check whether the value of the Resources parameter in the command output contains the name of the compressed package in the error message.
    If the name of the compressed package does not exist, run the create function <function_name> as <'package_to_class'> using <'resource_list'>; command to register the MaxCompute UDF again. Add the name of the compressed package to the resource_list parameter in the command.
    For more information about how to register a function, see Create a UDF.
  - Solution 2: On the MaxCompute client, run the desc resource <resource_name>; command. Check whether the value of the Type parameter in the command output is ARCHIVE.
    If the value of the Type parameter is not ARCHIVE, run the add archive <file_name>; command to upload the compressed package again.
    For more information about how to upload resources, see Add resources.
  - Solution 3: On the MaxCompute client, run the desc function <function_name>; command. Check whether the package name and file name extension in the value of the Resources parameter in the command output are consistent with the actual package name and file name extension.
    If the package name or file name extension in the command output is inconsistent with the actual package name or file name extension, run the add archive <file_name>; command to upload the compressed package again. Set the file_name parameter to a value that is consistent with the actual package name and file name extension.
  - Solution 4: Troubleshoot all the UDFs on which the job depends, including the UDFs on which a view depends. Check the projects to which the UDFs belong and the resources that correspond to the UDFs. If the resources that have the same name exist in different projects, we recommend that you modify the UDFs on which the job depends or modify the resource name.
Problem description 3: When I call a MaxCompute UDF in which get_cache_table(table_name) is configured, the following error message is reported: odps.distcache.DistributedCacheError: Table resource "xxx_table_name" not found.
- Possible causes:
  - Cause 1: The table does not exist. When you register the MaxCompute UDF, you do not specify a table in the command.
  - Cause 2: The resource type of the table is not TABLE.
- Solutions:
  - Solution 1: On the MaxCompute client, run the desc function <function_name>; command. Check whether the value of the Resources parameter in the command output contains the name of the table in the error message.
    If the name of the table does not exist, run the create function <function_name> as <'package_to_class'> using <'resource_list'>; command to register the MaxCompute UDF again. Add the name of the table to the resource_list parameter in the command.
    For more information about how to register a function, see Create a UDF.
  - Solution 2: On the MaxCompute client, run the desc resource <resource_name>; command. Check whether the value of the Type parameter in the command output is TABLE.
    If the value of the Type parameter is not TABLE, run the add table <table_name>; command to upload the table again.
    For more information about how to upload resources, see Add resources.
Problem description 4: When I call a MaxCompute UDF that references a third-party package, the following error message is reported: ImportError: No module named 'xxx'.
- Possible causes:
  - Cause 1: The resource type of the third-party package is not ARCHIVE.
  - Cause 2: When you register the MaxCompute UDF, you do not specify a third-party package in the command.
  - Cause 3: You do not specify the path of the third-party package in the MaxCompute UDF code.
  - Cause 4: The third-party package is a WHEEL file but its file name extension is invalid. You must download a WHEEL file based on the Python version.
  - Cause 5: The third-party package is not a WHEEL file or a pure Python package, but the package contains the setup.py file.
  - Cause 6: The name of the Python file for the MaxCompute UDF conflicts with the name of the third-party module that is referenced by the MaxCompute UDF. For example, if the Python file for the MaxCompute UDF is A.py, "import A" in the code imports the A.py file rather than the module in the third-party package.
- Solutions:
  - Solution 1: On the MaxCompute client, run the desc resource <resource_name>; command. Check whether the value of the Type parameter in the command output is ARCHIVE.
    If the value of the Type parameter is not ARCHIVE, run the add archive <file_name>; command to upload the compressed package again.
    For more information about how to upload resources, see Add resources.
  - Solution 2: On the MaxCompute client, run the desc function <function_name>; command. Check whether the value of the Resources parameter in the command output contains the name of the third-party package in the error message.
    If the name of the third-party package does not exist, run the create function <function_name> as <'package_to_class'> using <'resource_list'>; command to register the MaxCompute UDF again. Add the name of the third-party package to the resource_list parameter in the command.
    For more information about how to register a function, see Create a UDF.
  - Solution 3: Check whether the path of the third-party package is configured in sys.path.insert(0, 'work/third-party package path') in the MaxCompute UDF code. For example, the module name is A and the Python file is A.py. The following examples show you how to determine the path that you need to configure in the code:
    - Example 1: The Python file is located in the resource_dir folder, and this folder is compressed into the resource-of-A.zip package. The path that you need to configure in sys.path.insert is work/resource-of-A.zip/resource_dir/.
    - Example 2: The Python file is located in the resource_dir folder, and all files in this folder are compressed into the resource-of-A.zip package. The path that you need to configure in sys.path.insert is work/resource-of-A.zip/.
    - Example 3: The Python file is located in the resource_dir/path1/path2 folder and all files in the resource_dir folder are compressed into the resource-of-A.zip package. The path that you need to configure in sys.path.insert is work/resource-of-A.zip/path1/path2/.
    Note
    By default, resources of the ARCHIVE type are placed in the relative path ./work/ of the execution path of the MaxCompute UDF.
  - Solution 4: The WHEEL file varies based on the Python version. If you use Python 2, you must download the WHEEL file whose name contains cp27-cp27m-manylinux1_x86_64. If you use Python 3, you must download the WHEEL file whose name contains cp37-cp37m-manylinux1_x86_64. You can change the file name extension of the downloaded WHEEL file to .zip. This way, you do not need to package the WHEEL file into a ZIP file.
  - Solution 5: In a Python environment that is compatible with MaxCompute, compile the setup.py file into a WHEEL file. Then, upload resources and register the MaxCompute UDF. For more information about how to compile a third-party package, see Reference third-party packages that need to be compiled.
  - Solution 6: Change the name of the Python file for the MaxCompute UDF.
Problem description 5: When I call a MaxCompute UDF that references a standard Python 3 library, the following error message is reported: ImportError: No module named enum.
- Cause: Python 3 is not enabled for MaxCompute projects. By default, Python 2 is used to call MaxCompute UDFs. Therefore, the standard Python 3 library cannot be identified.
- Solution: Add the set odps.sql.python.version=cp37; command before the SQL statement that is used to call the MaxCompute UDF and submit the command together with the SQL statement.
Problem description 6: When I call a MaxCompute UDF, the following error message is reported: ModuleNotFoundError: No module named 'six'.
- Cause: When you use a third-party package together with a Python UDF, the path of the package is not added to sys.path. As a result, the third-party package cannot be imported.
- Solution: Change include_package_path('six.zip') to sys.path.insert(0, 'work/six.zip'). For more information, see Run Scipy in a MaxCompute UDF.
Problem description 7: When I call a MaxCompute UDF, the following error message is reported: failed to get Udf info from xxx.py.
- Cause: In the code of the user-defined table-valued function (UDTF) or user-defined aggregate function (UDAF), the statement used for importing the base class is invalid. For example, if you use import odps.udf.BaseUDTF or import odps.udf.BaseUDAF, the error message is reported.
- Solution: Change the statement to from odps.udf import BaseUDTF or from odps.udf import BaseUDAF.

Performance

Problem description: When I call a MaxCompute UDF, the following error message is reported: kInstanceMonitorTimeout.
Cause: Data processing of the MaxCompute UDF times out. By default, the duration in which the UDF processes data is limited. In most cases, a UDF must process 1024 rows of data at a time within 1800s. This duration is not the total duration in which a worker runs but the duration in which the UDF processes a small batch of data records at a time. In most cases, MaxCompute SQL can process more than 10,000 rows of data per second. This limit aims only to prevent infinite loops in a MaxCompute UDF. If an infinite loop occurs, CPU resources are occupied for a long period of time.

Solutions:

Add information about logging to the MaxCompute UDF code. This way, you can view the logs to check whether an infinite loop occurs. You can also check whether the duration of processing a single record by the MaxCompute UDF meets your expectation based on the time information recorded in the logs. Add the following information about logging to the code. After your job runs, you can view the logs in StdOut of the Logview UI.
- Python 2
```
sys.stdout.write('your log')
sys.stdout.flush()
```
- Python 3
```
print('your log', flush=True)
```

If the amount of data is large, the MaxCompute UDF may run for a long period of time. You can adjust the following parameters to prevent the timeout error.

Parameter	Description
`set odps.function.timeout=xxx;`	The timeout period of the MaxCompute UDF. Default value: 1800s. You can set this parameter to a larger value based on your business requirements. Valid values: 1s to 3600s.
`set odps.sql.executionengine.batch.rowcount=xxx;`	The number of data rows that the MaxCompute UDF processes at a time. Default value: 1024. You can set this parameter to a smaller value based on your business requirements.

Network

Problem description: When I call a MaxCompute UDF to access the Internet, an error is reported.
Cause: MaxCompute UDFs do not support access to the Internet.
Solution: Fill out and submit the network connection application form based on your business requirements. The MaxCompute technical support team then contacts you to establish a network connection. For more information about how to fill out the application form, see Network connection process.

Sandbox

Problem description: When I call a MaxCompute UDF, the following error message is reported: RuntimeError: xxx has been blocked by sandbox.
Cause: Some calls to Python UDFs are blocked by sandboxes.
Solutions:
- Add the set odps.isolation.session.enable=true; command before the SQL statement that is used to call a Python UDF and submit the command together with the SQL statement.
- By default, odps.isolation.session.enable is set to true for Python 3 UDFs.

Encoding

The following encoding problems may occur:

Problem description 1: When I call a MaxCompute UDF, the following error message is reported: SyntaxError: Non-ASCII character '\xe8' in file xxx. on line yyy.
- Cause: The Python file for the MaxCompute UDF contains non-ASCII characters and runs in Python 2.
- Solutions:
  - Add the set odps.sql.python.version=cp37; command before the SQL statement that is used to call the MaxCompute UDF and submit the command together with the SQL statement. This way, Python 3 is used to run jobs.
  - Add the following information to the beginning of the Python file. This way, the default encoding format of Python 2 is changed to UTF-8.
```
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
```
Problem description 2: When I call a MaxCompute UDF that is written in Python 2, the following error message is reported: UnicodeEncodeError: 'ascii' code can't encode characters in position x-y: ordinal not in range(128).
- Cause: The return value type that is specified in the function signature is STRING. However, the MaxCompute UDF returns a Python object of the UNICODE type. For example, if the Python object is named ret, MaxCompute uses str(ret) to convert the return value ret to a value of the STR type. If ret is within the ASCII encoding range, it can be converted to the STR type. However, if ret is not within the ASCII encoding range, the conversion fails and an error is reported.
- Solution: Add the following statement to the evaluate method in the Python code:
```
return ret.encode('utf-8')
```
Problem description 3: When I call a MaxCompute UDF that is written in Python 3, the following error message is reported: UnicodeDecodeError: 'utf-8' codec can't decode byte xxx in position xxx: invalid continuation byte.
- Cause: The input parameter type that is specified in the function signature is STRING. However, the string that is entered when you call the Python 3 UDF cannot be decoded in the UTF-8 format into a Python object of the STR type.
- Solutions:
  - Do not write strings that are encoded in a format other than UTF-8 to MaxCompute tables.
    A Python 2 UDF returns a GBK-based Python object that is of the STR type. The Python object can be normally written to a MaxCompute table but cannot be read by a Python 3 UDF. Therefore, we recommend that the Python object be encoded in the UTF-8 format before it is returned by the Python 2 UDF. For example, ret.decode('gbk').encode('utf-8') can be used to encode the Python object ret.
  - Use the built-in function is_encoding in an SQL statement to filter out the data that is encoded in a format other than UTF-8. Sample statement:
```
select py_udf(input_col) from example_table where is_encoding(input_col, 'utf-8', 'utf-8') = true;
```
  - Change the input parameter type that is specified in the function signature of the Python code to BINARY. Then, convert the data type of the columns that are of the STRING type to the BINARY type in the SQL statement and use the columns as the input parameters of the Python 3 UDF. Sample statement:
```
select py_udf(cast(input_col as binary)) from example_table;
```

Function signature

The following function signature problems may occur:

Problem description 1: When I call a MaxCompute UDF, the following error message is reported: resolve annotation of class xxx for UDTF/UDF/UDAF yyy contains invalid content '<EOF>'.
- Cause: The input or output parameters of a MaxCompute UDF are of a complex data type, and the related information in the function signature is invalid.
- Solution: Modify the information about the complex data type in the function signature to make sure that the function signature is valid. For more information about function signatures, see Function signatures and data types.
Problem description 2: When I call a MaxCompute UDF, the following error message is reported: TypeError: expected <class 'xxx'> but <class 'yyy'> found, value:zzz.
- Cause: The return value type that is specified in the function signature is inconsistent with the data type of the data that is returned by the MaxCompute UDF.
- Solution: Confirm the expected data type and modify the function signature or MaxCompute UDF code to ensure consistency.
Problem description 3: When I call a MaxCompute UDF, the following error message is reported: Semantic analysis exception - evaluate function in class xxx.yyy for user defined function zz does not match annotation ***->***.
- Cause: The number of input parameters that are specified in the function signature is inconsistent with the number of input parameters for the related method in the MaxCompute UDF code.
- Solution: Confirm the actual number of input parameters and modify the function signature or MaxCompute UDF code to ensure consistency.

Third-party packages

Problem description: When I call a MaxCompute UDF, the following error message is reported: GLIBCXX_x.x.x not found.
Cause: The GLIBCXX version on which the .so library file depends is later than the version that is supported by MaxCompute. The same issue may occur on glibc and CXXABI versions.
Solution: Use a compatible wheel package or recompile the .so library file in an environment that is compatible with MaxCompute. MaxCompute supports the following latest versions on which binary executable files or .so library files depend:
```
GLIBC <= 2.17
CXXABI <= 1.3.8
GLIBCXX <= 3.4.19
GCC <= 4.2.0
```

UDTFs

Problem description: When I call a MaxCompute UDTF, the following error message is reported: Semantic analysis exception - expect 2 aliases but have 0.
Cause: No output column name is specified in the Python UDTF code.
Solution: Use the AS clause to specify column names in the SELECT statement that is used to call the Python UDTF. Sample statement:
```
select my_udtf(col0, col1) as (ret_col0, ret_col1, ret_col2) from tmp1;
```

UDAFs

Problem description 1: When I call a MaxCompute UDAF, the following error message is reported: Script exception - ValueError: unmarshallable object.
- Cause: The value of the buffer parameter in the Python UDAF code is an unmarshallable object. For more information, see Marshal.
- Solution: When you assign a value to buffer, make sure that the value is a marshallable object. If you want to use two buffers that are of the LIST and DICT types in the Python UDAF, you must use return [list(), dict()] in the new_buffer method. When you use buffer or pbuffer in the iterate, merge, or terminate method, the buffer of the LIST type corresponds to buffer[0] or pbuffer[0], and the buffer of the DICT type corresponds to buffer[1] or pbuffer[1]. If the elements of a buffer are of the LIST or DICT type, the elements must also be marshallable objects.
Problem description 2: When I call a MaxCompute UDAF, the following error message is reported: Python UDAF buffer size overflowed: 2821486749.
- Cause: The buffer size in the Python UDAF code exceeds 2 GB after the marshaling operation. The buffer size is processed in an incorrect manner. As a result, the buffer size increases with the data amount.
- Solution: Rewrite the logic of the Python UDAF code. This way, the buffer size does not increase with the data amount. For example, if a buffer is a list, data cannot be added to the buffer in the iteration and merging phases. For more information about Python UDAFs, see Overview.