Container Service for Kubernetes:Use Triton to deploy Qwen inference services in ACK

Background

Prerequisites

Before you begin, make sure that you have:

• An ACK Pro cluster with GPU-accelerated nodes (Kubernetes 1.22 or later, 16 GB or more GPU memory per node). For more information, see Create an ACK managed cluster.

• GPU driver version 525 installed on GPU nodes. Add the ack.aliyun.com/nvidia-driver-version:525.105.17 label to GPU-accelerated nodes to pin the driver version. For more information, see Specify an NVIDIA driver version for nodes by adding a label.

• The latest Arena client installed. For more information, see Configure the Arena client.

Step 1: Prepare model data

Download the Qwen1.5-4B-Chat model, upload it to Object Storage Service (OSS), and create persistent volumes (PVs) and persistent volume claims (PVCs) in the ACK cluster.

To deploy other vLLM-supported models, see Supported models. To use File Storage NAS instead of OSS, see Mount a statically provisioned NAS volume.

Step 2: Configure Triton with vLLM

Create two configuration files: config.pbtxt for the Triton backend, and model.json for vLLM engine parameters.

mkdir triton-vllm

cat << EOF > triton-vllm/config.pbtxt
backend: "vllm"

# The usage of device is deferred to the vLLM engine
instance_group [
  {
    count: 1
    kind: KIND_MODEL
  }
]

version_policy: { all { }}
EOF

cat << EOF > triton-vllm/model.json
{
    "model":"/model/Qwen1.5-4B-Chat",
    "disable_log_requests": "true",
    "gpu_memory_utilization": 0.95,
    "trust_remote_code": "true",
    "max_model_len": 16384
}
EOF

cat << EOF > triton-vllm/model.json
{
    "model":"/model/Qwen1.5-4B-Chat",
    "disable_log_requests": "true",
    "gpu_memory_utilization": 0.95,
    "trust_remote_code": "true",
    "dtype": "half",
    "max_model_len": 8192
}
EOF

Step 3: Deploy the inference service

Use Arena to deploy the Qwen1.5-4B-Chat inference service with Triton and vLLM.

1. Export the configuration file paths as environment variables:

      export triton_config_file="triton-vllm/config.pbtxt"
      export model_config_file="triton-vllm/model.json"

2. Deploy the inference service:

   Parameters

   Parameter	Description
   --name	Name of the inference service.
   --version	Version of the inference service.
   --image	Container image for the Triton server.
   --gpus	Number of GPUs per replica.
   --cpu	Number of CPU cores per replica.
   --memory	Memory allocation per replica.
   --data	PVC mount in <pvc-name>:<mount-path> format. Run arena data list to list available PVCs.
   --config-file	Local file mount in <local-path>:<container-path> format.
   --model-repository	Triton model repository directory. Each subdirectory represents a model and must contain its configuration files. For more information, see Triton model repository.
   --http-port	HTTP port for the Triton service.
   --grpc-port	gRPC port for the Triton service.
   --allow-metrics	Expose inference metrics (GPU utilization, latency, throughput).

      arena serve triton \
          --name=triton-vllm \
          --version=v1 \
          --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/tritonserver:24.04-vllm-python-py3-ubuntu22.04 \
          --gpus=1 \
          --cpu=6 \
          --memory=30Gi \
          --data="llm-model:/model/Qwen1.5-4B-Chat" \
          --model-repository /triton-config \
          --config-file="$model_config_file:/triton-config/qwen-4b/1/model.json" \
          --config-file="$triton_config_file:/triton-config/qwen-4b/config.pbtxt" \
          --http-port=8000 \
          --grpc-port=9000 \
          --allow-metrics=true

   Expected output:

      configmap/triton-vllm-v1-4bd5884e6b5b6a3 created
      configmap/triton-vllm-v1-7815124a8204002 created
      service/triton-vllm-v1-tritoninferenceserver created
      deployment.apps/triton-vllm-v1-tritoninferenceserver created
      INFO[0007] The Job triton-vllm has been submitted successfully
      INFO[0007] You can run `arena serve get triton-vllm --type triton-serving -n default` to check the job status

3. Verify that the service is running. Wait until Available shows 1.

      arena serve get triton-vllm

   Expected output:

      Name:       triton-vllm
      Namespace:  default
      Type:       Triton
      Version:    v1
      Desired:    1
      Available:  1
      Age:        3m
      Address:    172.16.XX.XX
      Port:       RESTFUL:8000,GRPC:9000
      GPU:        1
   
      Instances:
        NAME                                                  STATUS   AGE  READY  RESTARTS  GPU  NODE
        ----                                                  ------   ---  -----  --------  ---  ----
        triton-vllm-v1-tritoninferenceserver-b69cb7759-gkwz6  Running  3m   1/1    0         1    cn-beijing.172.16.XX.XX

Step 4: Verify the inference service

(Optional) Clean up

Delete the inference service and storage resources when they are no longer needed:

# Delete the inference service
arena serve del triton-vllm

# Delete the PVC and PV
kubectl delete pvc llm-model
kubectl delete pv llm-model