Use an LLM as an inference service

Prerequisites

• The Model Service Mesh (hereinafter referred to as ModelMesh) feature is enabled in your Service Mesh (ASM) instance and the ASM environment is configured. For more information, see Step 1 and Step 2 in Use ModelMesh to roll out a multi-model inference service.

• You have learned how to use ModelMesh to create custom model serving runtimes. For more information, see Use ModelMesh to create a custom model serving runtime.

Step 1: Build a custom runtime

Build a custom runtime to serve the Hugging Face LLM with prompt tuning configuration. In this example, the default values are set to the pre-built custom runtime image and pre-built prompt tuning configuration.

1. Implement a class that inherits from the MLModel class of MLServer.

   The peft_model_server.py file contains all the code on how the Hugging Face LLM with prompt tuning configuration is being served. The _load_model function in the file is used to choose a pretrained LLM model with the PEFT prompt tuning configuration trained. The _load_model function also defines a tokenizer that can encode and decode raw string inputs from the inference requests without asking users to preprocess their input into tensor bytes.

   Show the peft_model_server.py file

   from typing import List
   
   from mlserver import MLModel, types
   from mlserver.codecs import decode_args
   
   from peft import PeftModel, PeftConfig
   from transformers import AutoModelForCausalLM, AutoTokenizer
   import torch
   import os
   
   class PeftModelServer(MLModel):
       async def load(self) -> bool:
           self._load_model()
           self.ready = True
           return self.ready
   
       @decode_args
       async def predict(self, content: List[str]) -> List[str]:
           return self._predict_outputs(content)
   
       def _load_model(self):
           model_name_or_path = os.environ.get("PRETRAINED_MODEL_PATH", "bigscience/bloomz-560m")
           peft_model_id = os.environ.get("PEFT_MODEL_ID", "aipipeline/bloomz-560m_PROMPT_TUNING_CAUSAL_LM")
           self.tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, local_files_only=True)
           config = PeftConfig.from_pretrained(peft_model_id)
           self.model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path)
           self.model = PeftModel.from_pretrained(self.model, peft_model_id)
           self.text_column = os.environ.get("DATASET_TEXT_COLUMN_NAME", "Tweet text")
           return
   
       def _predict_outputs(self, content: List[str]) -> List[str]:
           output_list = []
           for input in content:
               inputs = self.tokenizer(
                   f'{self.text_column} : {input} Label : ',
                   return_tensors="pt",
               )
               with torch.no_grad():
                   inputs = {k: v for k, v in inputs.items()}
                   outputs = self.model.generate(
                       input_ids=inputs["input_ids"], attention_mask=inputs["attention_mask"], max_new_tokens=10, eos_token_id=3
                   )
                   outputs = self.tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)
               output_list.append(outputs[0])
           return output_list
   

2. Build a Docker image.

   After the model class is implemented, you need to package its dependencies, including MLServer, into an image that is supported as a ServingRuntime resource. You can refer to the following Dockerfile to build an image:

   Show the Dockerfile

   # TODO: choose appropriate base image, install Python, MLServer, and
   # dependencies of your MLModel implementation
   FROM python:3.8-slim-buster
   RUN pip install mlserver peft transformers datasets
   # ...
   
   # The custom `MLModel` implementation should be on the Python search path
   # instead of relying on the working directory of the image. If using a
   # single-file module, this can be accomplished with:
   COPY --chown=${USER} ./peft_model_server.py /opt/peft_model_server.py
   ENV PYTHONPATH=/opt/
   
   # environment variables to be compatible with ModelMesh Serving
   # these can also be set in the ServingRuntime, but this is recommended for
   # consistency when building and testing
   ENV MLSERVER_MODELS_DIR=/models/_mlserver_models \
       MLSERVER_GRPC_PORT=8001 \
       MLSERVER_HTTP_PORT=8002 \
       MLSERVER_LOAD_MODELS_AT_STARTUP=false \
       MLSERVER_MODEL_NAME=peft-model
   
   # With this setting, the implementation field is not required in the model
   # settings which eases integration by allowing the built-in adapter to generate
   # a basic model settings file
   ENV MLSERVER_MODEL_IMPLEMENTATION=peft_model_server.PeftModelServer
   
   CMD mlserver start ${MLSERVER_MODELS_DIR}
   

3. Create a new ServingRuntime resource.

   1. Create a new ServingRuntime resource by using the following content and point it to the image you created.

      Show the YAML file

      apiVersion: serving.kserve.io/v1alpha1
      kind: ServingRuntime
      metadata:
        name: peft-model-server
        namespace: modelmesh-serving
      spec:
        supportedModelFormats:
          - name: peft-model
            version: "1"
            autoSelect: true
        multiModel: true
        grpcDataEndpoint: port:8001
        grpcEndpoint: port:8085
        containers:
          - name: mlserver
            image:  registry.cn-beijing.aliyuncs.com/test/peft-model-server:latest
            env:
              - name: MLSERVER_MODELS_DIR
                value: "/models/_mlserver_models/"
              - name: MLSERVER_GRPC_PORT
                value: "8001"
              - name: MLSERVER_HTTP_PORT
                value: "8002"
              - name: MLSERVER_LOAD_MODELS_AT_STARTUP
                value: "true"
              - name: MLSERVER_MODEL_NAME
                value: peft-model
              - name: MLSERVER_HOST
                value: "127.0.0.1"
              - name: MLSERVER_GRPC_MAX_MESSAGE_LENGTH
                value: "-1"
              - name: PRETRAINED_MODEL_PATH
                value: "bigscience/bloomz-560m"
              - name: PEFT_MODEL_ID
                value: "aipipeline/bloomz-560m_PROMPT_TUNING_CAUSAL_LM"
              # - name: "TRANSFORMERS_OFFLINE"
              #   value: "1"  
              # - name: "HF_DATASETS_OFFLINE"
              #   value: "1"    
            resources:
              requests:
                cpu: 500m
                memory: 4Gi
              limits:
                cpu: "5"
                memory: 5Gi
        builtInAdapter:
          serverType: mlserver
          runtimeManagementPort: 8001
          memBufferBytes: 134217728
          modelLoadingTimeoutMillis: 90000
      

   2. Run the following command to deploy the ServingRuntime resource:

      kubectl apply -f sample-runtime.yaml

      After you create the ServingRuntime resource, you can see the new custom runtime in your ModelMesh deployment.

Step 2: Deploy an LLM service

To deploy a model by using the newly created runtime, you must create an InferenceService resource to serve the model. This resource is the main interface used by KServe and ModelMesh to manage models. It represents the logical endpoint of the model for serving inferences.

1. Create an InferenceService resource to serve the model by using the following content:

   Show the YAML file

   apiVersion: serving.kserve.io/v1beta1
   kind: InferenceService
   metadata:
     name: peft-demo
     namespace: modelmesh-serving
     annotations:
       serving.kserve.io/deploymentMode: ModelMesh
   spec:
     predictor:
       model:
         modelFormat:
           name: peft-model
         runtime: peft-model-server
         storage:
           key: localMinIO
           path: sklearn/mnist-svm.joblib
   

   In the YAML file, the InferenceService resource is named peft-demo and its model format is declared as peft-model, which is the same format as the example custom runtime created in the previous step. An optional field runtime is also passed, explicitly instructing ModelMesh to use the peft-model-server runtime to deploy this model.

2. Run the following command to deploy the InferenceService resource:

   kubectl apply -f ${Name of the YAML file}.yaml

Step 3: Perform an inference

Run the curl command to send an inference request to the LLM service deployed in the previous step.

MODEL_NAME="peft-demo"
ASM_GW_IP="IP address of the ingress gateway"
curl -X POST -k http://${ASM_GW_IP}:8008/v2/models/${MODEL_NAME}/infer -d @./input.json

input.json in the curl command indicates the request data:

{
    "inputs": [
        {
          "name": "content",
          "shape": [1],
          "datatype": "BYTES",
          "contents": {"bytes_contents": ["RXZlcnkgZGF5IGlzIGEgbmV3IGJpbm5pbmcsIGZpbGxlZCB3aXRoIG9wdGlvbnBpZW5pbmcgYW5kIGhvcGU="]}
        }
    ]
}

The value of bytes_contents is the Base64 encoded content of the string "Every day is a new beginning, filled with opportunities and hope".

The following code block shows the JSON response:

{
 "modelName": "peft-demo__isvc-5c5315c302",
 "outputs": [
  {
   "name": "output-0",
   "datatype": "BYTES",
   "shape": [
    "1",
    "1"
   ],
   "parameters": {
    "content_type": {
     "stringParam": "str"
    }
   },
   "contents": {
    "bytesContents": [
     "VHdlZXQgdGV4dCA6IEV2ZXJ5IGRheSBpcyBhIG5ldyBiaW5uaW5nLCBmaWxsZWQgd2l0aCBvcHRpb25waWVuaW5nIGFuZCBob3BlIExhYmVsIDogbm8gY29tcGxhaW50"
    ]
   }
  }
 ]
}

The following code block shows the Base64-decoded content of bytesContents. It indicates that the inference request is performed on the LLM service as expected.

Tweet text : Every day is a new binning, filled with optionpiening and hope Label : no complaint