By Alex Mungai Muchiri, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.
MXNet is a great tool to use as your deep learning engine deployed in the cloud. MXNet provides a reliable tool to train your machine learning (ML) models, which can then be imported to portable devices such as smartphones. Furthermore, the community provides numerous resources to support the project. Some of the things to love about MXNet include:
For most first-time machine learning developers, Keras is a good choice. It is also easier to use it as the foundation for a neural network architecture as well as the training phases. However, MXNet allows massive scalability from those foundational experiments to ImageNet-size datasets. With MXNet, you can build datasets that are packed efficiently and have the network trained on multiple GPUs on several machines. Furthermore, it is possible to have the system performance in the optimal because C/C++ binaries bind Python to MXNet. MXNet being an Apache project has numerous developers creating new tools and tutorials for the community. Therefore, MXNet is a great choice for your deep learning backend.
This tutorial teaches several ways you can install MXNet, the deep learning tool on Alibaba Cloud ECS running Ubuntu 16.04. At the time of writing, MXNet does not support a higher version for Ubuntu. In particular, this series is meant to help developers to create and train new deep learning models using MXNet.
MXNet enables building with GPU, which requires setting up the runtime for CUDA and cuDNN. To begin with, install the CUDA toolkit after downloading it. The recommended version is CUDA 9.2. Then, download cuDNN 7.1.4 and unzip the file.
Then, in the cuDNN root directory, place its header and libraries in the local CUDA Toolkit folder.
tar xvzf cudnn-9.2-linux-x64-v7.1
sudo cp -P cuda/include/cudnn.h /usr/local/cuda/include
sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
sudo ldconfigRun these scripts to install Ubuntu 16.04 dependencies to support MXNet Python development.
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_core.sh
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_python.sh
sudo ./ubuntu_core.sh
sudo ./ubuntu_python.shIn order to achieve the fastest training speed with the tool, it is best to use the latest MXNet with CUDA 9.2 package.
Install training package:
pip install mxnet-cu92Install inference package:
pip install mxnet-cu92mklHowever, you can select a different combination if your objectives are not merely the training speed.
In this table, we list pip packages recommended for respective MXNet versions.
When installing any MXNet version coupled with one of the packages listed in the table, include == in the required version. For instance, to install MXNet 1.2.0 with CUDA 8, run the command:
pip install mxnet-cu80==1.2.0.There is also the option of building MXNet from source, in which case you can easily install builds that are specific to supported languages such as Scala, Julia, R or Perl. There are two steps involved in the process:
Notably, you can change your build's compilation options using the make/config.mk file before building MXNet.
There are two options for building the MXNet library: quick or manual.
For the quick build of MXNet, you will find the script below in the /docs/install folder:
cd docs/install
./install_mxnet_ubuntu_python.shInstall git and the build tools.
sudo apt-get update
sudo apt-get install -y build-essential gitInstall OpenBLAS. A BLAS library helps to accelerate MXNet numerical computations on CPU machines. It comes in different versions including OpenBLAS, ATLAS, and MKL. While you can choose either of the libraries, we shall go with OpenBLAS for now:
sudo apt-get install -y libopenblas-devInstall OpenCV. OpenCV for MXNet enables the loading of images and augmenting operations efficiently.
sudo apt-get install -y libopencv-devBuild an MXNet core shared library from downloaded MXNet sources. There is option for building on CPU or GPU
Build on CPU:
git clone --recursive https://github.com/apache/incubator-mxnet.git
cd mxnet
make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblasTo build on GPU, verify you have all CUDA dependencies installed):
git clone --recursive https://github.com/apache/incubator-mxnet.git
cd mxnet
make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1The USE_OPENCV and USE_BLAS file flags rely on OpenCV and BLAS library.
There are further compilation options in make/config.mk. once executed, the commands create a library named libmxnet.so.
If you need a library for visualizing network graphs on MXNet, you can additionally install graphviz.
Jupyter Notebook is another library you can install to run tutorials:
sudo apt-get install -y python-pip
sudo pip install graphviz
sudo pip install jupyterAfter installing the core library, install the preferred programming language interface package of your choice: Scala, R, Julia, Perl.
For this package, you will need the Maven package as a dependency. Run the scripts below:
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_core.sh
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_scala.sh
sudo ./ubuntu_core.sh
sudo ./ubuntu_scala.shNext, run the MXNet-Scala demo project. The project is available on the open source MXNet-Scala demo project's README.
Run the Maven repository like so:
Linux CPU
<!-- https://mvnrepository.com/artifact/org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu -->
<dependency>
<groupId>org.apache.mxnet</groupId>
<artifactId>mxnet-full_2.11-linux-x86_64-cpu</artifactId>
</dependency>Linux GPU
<!-- https://mvnrepository.com/artifact/org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu -->
<dependency>
<groupId>org.apache.mxnet</groupId>
<artifactId>mxnet-full_2.11-linux-x86_64-gpu</artifactId>
</dependency>There is also the option for running the R package, which is a two-step process:
libmxnet.so from source, which is an MXNet core shared libraryThere is the option for either quick or manual installation:
Quick Installation
For a quick MXNet-R installation, run the scripts below:
git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet
cd mxnet/docs/install
./install_mxnet_ubuntu_python.sh
./install_mxnet_ubuntu_r.shManual MXNet-R Installation
The manual process is an alternative, but there are minimum requirements:
Build the MXNet core shared library by installing the tools:
$ sudo apt-get update
$ sudo apt-get install -y build-essential gitInstall OpenBLAS. We shall select ATLAS or MKL but in this case, we choose OpenBLAS.
$ sudo apt-get install -y libopenblas-dev liblapack-devInstall OpenCV like so:
$ sudo apt-get install -y libopencv-devBuild an MXNet core shared library after downloading the source.
$ git clone --recursive https://github.com/apache/incubator-mxnet
$ cd incubator-mxnet
$ make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblasMake the MXNet-R bindings and then install them like so:
$ make rpkgRun the command below to verify your MXNet-R installation:
sudo -i RWhen prompted with an R prompt respond as so:
library(mxnet)
a <- mx.nd.ones(c(2,3), ctx = mx.cpu())
b <- a * 2 + 1
bThe output should be similar:
[,1] [,2] [,3]
[1,] 3 3 3
[2,] 3 3 3
> quit()The Julia MXNet package is available on GitHub but ins separate repository names MXNet.jl. However, the package requires that you set the MXNET_HOME environment variable to bind Julia to a libmxnet installation. Run the command below:
export MXNET_HOME=/<path to>/libmxnetThe root directory of libmxnet should have a path to the libmxnet installation on the machine. That is to mean, that $MXNET_HOME/lib contains the libmxnet.so file available. The requirement is that in the root directory of libmxnet, say ~, run the following:
export MXNET_HOME=/~/libmxnetAdditionally, you could also include the command in your ~/.bashrc file so as to install Julia package in the console like so:
Pkg.add("MXNet")If you are keen on learning more about Julia, check out the MXNet Julia documentation.
The MXNet-Scala package is another package that you could install, depending on your skill. With it, acquire Maven package as a dependency. On your Ubuntu machine, run the following:
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_core.sh
wget https://raw.githubusercontent.com/apache/incubator-mxnet/master/ci/docker/install/ubuntu_scala.sh
sudo ./ubuntu_core.sh
sudo ./ubuntu_scala.shRun the Maven repository like so:
Linux CPU
<!-- https://mvnrepository.com/artifact/org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu -->
<dependency>
<groupId>org.apache.mxnet</groupId>
<artifactId>mxnet-full_2.11-linux-x86_64-cpu</artifactId>
</dependency>Linux GPU
<!-- https://mvnrepository.com/artifact/org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu -->
<dependency>
<groupId>org.apache.mxnet</groupId>
<artifactId>mxnet-full_2.11-linux-x86_64-gpu</artifactId>
</dependency>Setting up for Scala was successful!
For Perl users, the first requirement is to must complete building the shared library. Thereafter, you can then build MXNet from its source. In the MXNet root directory, run the command below:
sudo apt-get install libmouse-perl pdl cpanminus swig libgraphviz-perl
cpanm -q -L "${HOME}/perl5" Function::Parameters Hash::Ordered PDL::CCS
MXNET_HOME=${PWD}
export LD_LIBRARY_PATH=${MXNET_HOME}/lib
export PERL5LIB=${HOME}/perl5/lib/perl5
cd ${MXNET_HOME}/perl-package/AI-MXNetCAPI/
perl Makefile.PL INSTALL_BASE=${HOME}/perl5
make install
cd ${MXNET_HOME}/perl-package/AI-NNVMCAPI/
perl Makefile.PL INSTALL_BASE=${HOME}/perl5
make install
cd ${MXNET_HOME}/perl-package/AI-MXNet/
perl Makefile.PL INSTALL_BASE=${HOME}/perl5
make installWe have looked at the various methods of installing MXNet and its dependencies on an Alibaba Cloud Elastic Compute Service (ECS) Ubuntu 16.04 server. Installation of MXNet is the first step for the deep learning project running on either CPU, GPU or both. Once installed, you would easily identify with the binaries that enable you to build image record files efficiently.
Alex - February 14, 2020
Alibaba Cloud Community - April 3, 2024
Alibaba Clouder - November 27, 2019
Alibaba Clouder - April 23, 2019
Alibaba Clouder - June 13, 2019
Alibaba Clouder - August 2, 2018
Platform For AI
A platform that provides enterprise-level data modeling services based on machine learning algorithms to quickly meet your needs for data-driven operations.
Learn More
Super App Solution for Telcos
Alibaba Cloud (in partnership with Whale Cloud) helps telcos build an all-in-one telecommunication and digital lifestyle platform based on DingTalk.
Learn More
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn More
EMAS Superapp
Build superapps and corresponding ecosystems on a full-stack platform
Learn MoreMore Posts by Alex
