A processor is a package of online prediction logic. Elastic Algorithm Service (EAS) of Platform for AI (PAI) provides built-in processors, which are commonly used to deploy models. The built-in processors can help you reduce the costs of developing the online prediction logic of models.
The following table describes the names and codes of the processors provided by EAS. If you use the EASCMD client to deploy a model, a processor code is required.
Processor name | Processor code (required if EASCMD is used) | Reference |
CPU edition | GPU edition |
PMML processor
The built-in Predictive Model Markup Language (PMML) processor in EAS performs the following operations:
The PMML processor provides a default policy to fill in missing values. If the isMissing policy is not specified for the feature columns in the PMML file, the values in the following table are automatically used.
Data type | Default input value |
Data type | Default input value |
BOOLEAN | false |
DOUBLE | 0.0 |
FLOAT | 0.0 |
INT | 0 |
STRING | "" |
You can deploy a model from a PMML file by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to PMML. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to pmml. Sample code:
{
"processor": "pmml",
"generate_token": "true",
"model_path": "http://xxxxx/lr.pmml",
"name": "eas_lr_example",
"metadata": {
"instance": 1,
"cpu": 1 # Allocate 4 GB of memory to each CPU. One CPU and 4 GB memory are considered one quota.
}
}
Use Data Science Workshop (DSW) to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW.
TensorFlow1.12 Processor
The TensorFlow1.12 processor of EAS can load TensorFlow models in the SavedModel or SessionBundle format. We recommend that you use the SavedModel format. Before you can deploy the model, you must convert a Keras or Checkpoint model to a SavedModel model. For more information, see Export TensorFlow models in the SavedModel format.
Note
The general-purpose processor does not support custom TensorFlow operations.
You can deploy a TensorFlow model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to TensorFlow1.12. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to tensorflow_cpu_1.12 or tensorflow_gpu_1.12 based on the model resources. If the value of the processor parameter does not match the resource type, a deployment error occurs. Sample code:
{
"name": "tf_serving_test",
"generate_token": "true",
"model_path": "http://xxxxx/savedmodel_example.zip",
"processor": "tensorflow_cpu_1.12",
"metadata": {
"instance": 1,
"cpu": 1,
"gpu": 0,
"memory": 2000
}
}
Use DSW to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW.
TensorFlow1.14 Processor
The TensorFlow1.14 processor of EAS can load TensorFlow models in the SavedModel or SessionBundle format. We recommend that you use the SavedModel format. Before you can deploy the model, you must convert a Keras or Checkpoint model to a SavedModel model. For more information, see Export TensorFlow models in the SavedModel format.
Note
The general-purpose processor does not support custom TensorFlow operations.
You can deploy a TensorFlow model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to TensorFlow1.14. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to tensorflow_cpu_1.14 or tensorflow_gpu_1.14 based on the model resources. If the value of the processor parameter does not match the resource type, a deployment error occurs. Sample code:
{
"name": "tf_serving_test",
"generate_token": "true",
"model_path": "http://xxxxx/savedmodel_example.zip",
"processor": "tensorflow_cpu_1.14",
"metadata": {
"instance": 1,
"cpu": 1,
"gpu": 0,
"memory": 2000
}
}
Use DSW to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW.
TensorFlow1.15 processor with a built-in optimization engine based on PAI-Blade of the agility edition
The TensorFlow1.15 processor of EAS can load TensorFlow models in the SavedModel or SessionBundle format. We recommend that you use the SavedModel format. Before you can deploy the model, you must convert a Keras or Checkpoint model to a SavedModel model. For more information, see Export TensorFlow models in the SavedModel format.
Note
The general-purpose processor does not support custom TensorFlow operations.
TensorFlow1.15 processor provides a built-in optimization engine based on PAI-Blade of the agility edition. You can use this processor to deploy TensorFlow models that are optimized by PAI-Blade of the agility edition.
You can deploy a TensorFlow model by using one of the following methods:
Upload the model file to the console
Set the Processor Type parameter to TensorFlow1.15. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to tensorflow_cpu_1.15 or tensorflow_gpu_1.15 based on the model resources. If the value of the processor parameter does not match the resource type, a deployment error occurs. Sample code:
{
"name": "tf_serving_test",
"generate_token": "true",
"model_path": "http://xxxxx/savedmodel_example.zip",
"processor": "tensorflow_cpu_1.15",
"metadata": {
"instance": 1,
"cpu": 1,
"gpu": 0,
"memory": 2000
}
}
Use DSW to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW. For more information about the parameters in the service configuration file, see Create a service.
TensorFlow2.3 Processor
The TensorFlow2.3 processor EAS can load TensorFlow models in the SavedModel or SessionBundle format. We recommend that you use the SavedModel format. Before you can deploy the model, you must convert a Keras or Checkpoint model to a SavedModel model. For more information, see Export TensorFlow models in the SavedModel format.
Note
The general-purpose processor does not support custom TensorFlow operations.
You can deploy a TensorFlow model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to TensorFlow2.3. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to tensorflow_cpu_2.3. Sample code:
{
"name": "tf_serving_test",
"generate_token": "true",
"model_path": "http://xxxxx/savedmodel_example.zip",
"processor": "tensorflow_cpu_2.3",
"metadata": {
"instance": 1,
"cpu": 1,
"gpu": 0,
"memory": 2000
}
}
Use DSW to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW.
PyTorch1.6 processor with a built-in optimization engine based on PAI-Blade of the agility edition
The PyTorch1.6 processor of EAS can load models in the TorchScript format. For more information, see TorchScript.
Note
The general-purpose processor does not support PyTorch extensions. You cannot use this processor to import or export models other than TensorFlow models.
The PyTorch1.6 processor provides a built-in optimization engine based on PAI-Blade of the agility edition. You can use this processor to deploy PyTorch models that are optimized by PAI-Blade of the agility edition.
You can deploy a TorchScript model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to PyTorch1.6. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to pytorch_cpu_1.6 or pytorch_gpu_1.6 based on the model resources. If the value of the processor parameter does not match the resource type, a deployment error occurs. Sample code:
{
"name": "pytorch_serving_test",
"generate_token": "true",
"model_path": "http://xxxxx/torchscript_model.pt",
"processor": "pytorch_gpu_1.6",
"metadata": {
"instance": 1,
"cpu": 1,
"gpu": 1,
"cuda": "10.0",
"memory": 2000
}
}
Use DSW to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW. For more information about the parameters in the service configuration file, see Create a service.
Caffe Processor
The Caffe processor of EAS can load deep learning models that are trained based on the Caffe framework. The Caffe framework has flexible capabilities. When you deploy a Caffe model, you must specify the names of the model file and weight file in the model package.
Note
The general-purpose processor does not support custom data layers.
You can deploy a Caffe model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to Caffe. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to caffe_cpu or caffe_gpu based on the model resources. If the value of the processor parameter does not match the type of resource, a deployment error occurs. Sample code:
{
"name": "caffe_serving_test",
"generate_token": "true",
"model_path": "http://xxxxx/caffe_model.zip",
"processor": "caffe_cpu",
"model_config": {
"model": "deploy.prototxt",
"weight": "bvlc_reference_caffenet.caffemodel"
},
"metadata": {
"instance": 1,
"cpu": 1,
"gpu": 0,
"memory": 2000
}
}
Use DSW to deploy the model
Modify the service.json service configuration file. This method is similar to the method of deploying models by using the EASCMD client. For more information, see Use DSW.
PS processor
The PS processor of EAS is developed based on PS algorithms. The processor can load models in the PS format.
The following section describes how to deploy a PS model as a service and send requests by using a PS model service.
EasyTransfer Processor
The EasyTransfer processor of EAS can load TensorFlow-based deep learning natural language processing (NLP) models that are trained based on the EasyTransfer framework.
You can deploy an EasyTransfer model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to EasyTransfer. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to easytransfer_cpu or easytransfer_gpu based on the model resources. If the value of the processor parameter does not match the resource types, a deployment error occurs. Set the type field of the model_config parameter to the model type that you want to use. In the following examples, a text classification model is used. For more information about other parameters, see Create a service.
The following table lists the supported model types.
Job Type | Model type |
Text matching | text_match_bert |
Text classification | text_classify_bert |
Sequence labeling | sequence_labeling_bert |
Text vectorization | vectorization_bert |
EasyNLP Processor
The EasyNLP processor of EAS can load PyTorch-based deep learning NLP models that are trained based on the EasyNLP framework.
You can deploy an EasyNLP model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to EasyNLP. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to easynlp. Set the type field of the model_config parameter to the model type that you want to use. In the following examples, a single-label text classification model is used. For more information about other parameters, see Create a service.
{
"name": "easynlp_app_demo",
"metadata": {
"instance": 1
},
"cloud": {
"computing": {
"instance_type": "ecs.gn6i-c4g1.xlarge"
}
},
"model_config": {
"app_name": "text_classify",
"type": "text_classify"
},
"model_path": "http://xxxxx/your_model.tar.gz",
"processor": "easynlp"
}
The following table lists the supported model types.
Job Type | Model type |
Single-label text classification | text_classify |
Multi-label text classification | text_classify_multi |
Text matching | text_match |
Sequence labeling | sequence_labeling |
Text vectorization | vectorization |
Summary generation for Chinese text (GPU) | sequence_generation_zh |
Summary generation for English text (GPU) | sequence_generation_en |
Machine reading comprehension for Chinese text | machine_reading_comprehension_zh |
Machine reading comprehension for English text | machine_reading_comprehension_en |
WUKONG_CLIP (GPU) | wukong_clip |
CLIP (GPU) | clip |
After the model service is deployed to EAS, go to the Elastic Algorithm Service (EAS) page, find the service and click Invocation Method in the Service Type column to obtain the service endpoint and token. The following sample code provides a sample Python request that is used to call the service:
import requests
url = '<eas-service-url>'
token = '<eas-service-token>'
request_body = {
"first_sequence": "hello"
}
headers = {"Authorization": token}
resp = requests.post(url=url, headers=headers, json=request_body)
print(resp.content.decode())
EasyCV Processor
The EasyCV processor of EAS can load deep learning models that are trained based on the EasyCV framework.
You can deploy an EasyCV model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to EasyCV. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to easycv. Set the type field of the model_config parameter to the model type that you want to use. In the following examples, an image classification model is used. For more information about other parameters, see Create a service.
{
"name": "easycv_classification_example",
"processor": "easycv",
"model_path": "oss://examplebucket/epoch_10_export.pt",
"model_config": {"type":"TorchClassifier"},
"metadata": {
"instance": 1
},
"cloud": {
"computing": {
"instance_type": "ecs.gn5i-c4g1.xlarge"
}
}
}
The following table lists the supported model types.
Job Type | model_config |
Image classification | {"type":"TorchClassifier"} |
Object detection | {"type":"DetectionPredictor"} |
Semantic segmentation | {"type":"SegmentationPredictor"} |
YOLOX | {"type":"YoloXPredictor"} |
Video classification | {"type":"VideoClassificationPredictor"} |
After the model service is deployed to EAS, go to the Elastic Algorithm Service (EAS) page, find the service and click Invocation Method in the Service Type column to obtain the service endpoint and token. The following sample code provides a sample Python request that is used to call the service:
import requests
import base64
import json
resp = requests.get('http://exmaplebucket.oss-cn-zhangjiakou.aliyuncs.com/images/000000123213.jpg')
ENCODING = 'utf-8'
datas = json.dumps( {
"image": base64.b64encode(resp.content).decode(ENCODING)
})
head = {
"Authorization": "NTFmNDJlM2E4OTRjMzc3OWY0NzI3MTg5MzZmNGQ5Yj***"
}
for x in range(0,10):
resp = requests.post("http://150231884461***.cn-hangzhou.pai-eas.aliyuncs.com/api/predict/test_easycv_classification_example", data=datas, headers=head)
print(resp.text)
You must convert the images and video files to the Base64 format for transmission. Use image to specify image data and video to specify video data.
EasyVision Processor
The EasyVision processor of EAS can load deep learning models that are trained based on the EasyVision framework.
You can deploy an EasyVision model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to EasyVision. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy the model
In the service.json service configuration file, set the processor parameter to easy_vision_cpu_tf1.12_torch151 or easy_vision_gpu_tf1.12_torch151 based on the model resources. If the value of the processor parameter does not match the type of resource, a deployment error occurs. Set the type parameter in the model_config section to the type of the model that is trained. The following code block shows an example. For more information about other parameters, see Create a service.
MediaFlow Processor
The MediaFlow processor of EAS is a general-purpose orchestration engine that can analyze and process video, audio, and images.
You can deploy a MediaFlow model by using one of the following methods:
Upload the model file in the console
Set the Processor Type parameter to MediaFlow. For more information, see Upload and deploy models in the console.
Use the EASCMD client to deploy models
In the service.json service configuration file, set the processor parameter to mediaflow. If you use the MediaFlow processor to deploy models, you must configure the following parameters. For more information about other parameters, see Create a service.
Sample code:
In the service.json service configuration file, the values of the model_entry, name, and model_path parameters for video classification and ASR vary. You must configure the parameters based on the purpose of the model.
Triton Processor
Triton Inference Server is a new-generation online service framework released by NVIDIA. Triton Inference Server simplifies the deployment and management of GPU-accelerated models and complies with the API standards of KFServing. Triton Inference Server provides the following features:
Supports multiple open source frameworks such as TensorFlow, PyTorch, ONNX Runtime, TensorRT, and custom framework backends.
Concurrently runs multiple models on one GPU to maximize GPU utilization.
Supports the HTTP and gRPC protocols and allows you to send requests in binary format to reduce the request size.
Supports the dynamic batching feature to improve service throughput.
EAS provides a built-in Triton processor.
Note
The Triton processor is available for public preview only in the China (Shanghai) region.
The models that are deployed by using the Triton processor must be stored in Object Storage Service (OSS). Therefore, you must activate OSS and upload model files to OSS before you can use the Triton processor to deploy models. For information about how to upload objects to OSS, see Upload objects.
The following section describes how to use the Triton processor to deploy a model as a service and call the service: