How to Build a Crcmod on MaxCompute

By Ji Sheng

For binary packages, MaxCompute requires wheel packages with cp27-cp27m contained in their package names. However, for packages that have not been updated for a long time, for example, the crcmod package on which oss2 depends, PyPI does not provide wheel packages. Therefore, you have to create wheel packages yourself. This article describes how to use the quay.io/pypa/manylinux1_x86_64 image to build a wheel package that can be used on MaxCompute.

This article uses https://github.com/pypa/manylinux as a reference. The quay.io/pypa/manylinux1_x86_64 image is also the standard tool that the majority of Python projects currently use for packaging on Travis CI. If you have any further questions, you can study this project.

Prepare Dependencies

Many packages have dependencies, such as the devel rpm package or other Python packages. You need to know the dependencies of a specific package before packaging. Information about installation and packaging can usually be found on GitHub. The crcmod package has no other dependencies except gcc. Therefore, you can skip this step.

Modify and Verify setup.py (Recommended on Mac OS or Linux)

Older Python packages generally do not support building wheel packages. Specifically, an error is reported when 'python setup.py bdist_wheel' is used for packaging. To build wheel packages, you need to modify setup.py. For some packages, you can simply replace the setup function in distutils with the setup function in setuptools. For packages that require many custom operations in setup.py, you need to analyze the packaging process in detail. This is complicated and thus not discussed in this article.

For example, for crcmod, change

from distutils.core import setup in setup.py

from setuptools import setup

After making the preceding change, run the following in the root directory of the project:

python setup.py bdist_wheel

If no errors are reported and the generated wheel package can be used locally, it indicates that setup.py is available.

Prepare the Packaging Script

Create a bin directory in the project and build-wheel.sh under that directory:

mkdir bin && vim bin/build-wheel.sh

Enter the following content:

#! /bin/bash
# modified from https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh
set -e -x 
 
# Install a system package required by our library
# Change to the dependency installation command

# Compile wheels
PYBIN=/opt/python/cp27-cp27m/bin
# If dev-requirements.txt is present under the root directory of the package, remove the following comments.
# "${PYBIN}/pip" install -r /io/dev-requirements.txt
"${PYBIN}/pip" wheel /io/ -w wheelhouse/
 
# Bundle external shared libraries into the wheels
for whl in wheelhouse/*.whl; do
 auditwheel repair "$whl" -w /io/wheelhouse/
done

Enter the dependency installation script that you obtained in Step 1 into this script. Make sure to use the version in /opt/python/cp27-cp27m/bin when using Python or pip.

Finally, set the execute permission.

chmod a+x bin/build-wheel.sh

Packaging

Use Docker to download the required image and package the root directory of the project (Docker is required in this step. Please install it in advance):

docker pull quay.io/pypa/manylinux1_x86_64
docker run --rm -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/bin/build-wheel.sh

The generated wheel package is in the wheelhouse directory under the root directory of the project.

Community

How to Build a Crcmod on MaxCompute

Prepare Dependencies

Modify and Verify setup.py (Recommended on Mac OS or Linux)

Prepare the Packaging Script

Packaging

Read previous post:

Read next post:

Alibaba Cloud MaxCompute

You may also like

Comments

Alibaba Cloud MaxCompute

Related Products

Big Data Consulting for Data Technology Solution

MaxCompute

Big Data Consulting Services for Retail Solution

ApsaraDB for HBase