Container Service for Kubernetes:Use vLLM to deploy a Qwen model as an inference service

Qwen1.5-4B-Chat

Qwen1.5-4B-Chat is a 4-billion-parameter LLM developed by Alibaba Cloud based on the Transformer architecture. It is trained on large-scale data that covers web text, domain-specific books, and code. For more information, see the Qwen GitHub repository.

vLLM

vLLM is an open-source LLM inference framework that delivers high throughput and low latency. It supports most widely used LLMs, including Qwen models. vLLM uses PagedAttention optimization, continuous batching, and model quantization to improve inference efficiency. For more information, see the vLLM GitHub repository.

Download the model

1. Install Git:

      # Run yum install git or apt install git.
      yum install git

2. Install the Git Large File Storage (Git LFS) plug-in:

      # Run yum install git-lfs or apt install git-lfs.
      yum install git-lfs

3. Clone the Qwen1.5-4B-Chat repository from ModelScope:

      GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/qwen/Qwen1.5-4B-Chat.git

4. Enter the Qwen1.5-4B-Chat directory and pull the large files managed by Git LFS:

      cd Qwen1.5-4B-Chat
      git lfs pull

Upload the model to OSS

1. Log on to the OSS console and note the name of your OSS bucket. To create a bucket, see Create a bucket.

2. Install and configure ossutil. For more information, see Install ossutil.

3. Create a directory named Qwen1.5-4B-Chat in your OSS bucket:

      ossutil mkdir oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat

4. Upload the model files to OSS:

      ossutil cp -r ./Qwen1.5-4B-Chat oss://<Your-Bucket-Name>/Qwen1.5-4B-Chat

Create a PV and PVC

Create a PV and PVC in your ACK cluster to mount the model stored in OSS. For detailed instructions, see Mount a statically provisioned OSS volume.

Configure the PV with the following parameters:

Configure the PVC with the following parameters:

Choose a GPU type

Deploy the service

Choose one of the following commands based on your GPU type.

arena serve custom \
    --name=vllm-qwen-4b-chat \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --restful-port=8000 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/vllm:0.4.1-ubuntu22.04 \
    --data=llm-model:/model/Qwen1.5-4B-Chat \
    "python3 -m vllm.entrypoints.openai.api_server --trust-remote-code --model /model/Qwen1.5-4B-Chat/ --gpu-memory-utilization 0.95 --max-model-len 16384"

arena serve custom \
    --name=vllm-qwen-4b-chat \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --restful-port=8000 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --image=ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/vllm:0.4.1-ubuntu22.04 \
    --data=llm-model:/model/Qwen1.5-4B-Chat \
    "python3 -m vllm.entrypoints.openai.api_server --trust-remote-code --model /model/Qwen1.5-4B-Chat/ --gpu-memory-utilization 0.95 --max-model-len 8192 --dtype half"

The following table describes the Arena parameters:

Expected output:

service/vllm-qwen-4b-chat-v1 created
deployment.apps/vllm-qwen-4b-chat-v1-custom-serving created
INFO[0008] The Job vllm-qwen-4b-chat has been submitted successfully
INFO[0008] You can run `arena serve get vllm-qwen-4b-chat --type custom-serving -n default` to check the job status

Verify the deployment

Run the following command to check the status of the inference service:

arena serve get vllm-qwen-4b-chat

Expected output:

Name:       vllm-qwen-4b-chat
Namespace:  default
Type:       Custom
Version:    v1
Desired:    1
Available:  1
Age:        36m
Address:    172.16.XX.XX
Port:       RESTFUL:8000
GPU:        1

Instances:
  NAME                                                  STATUS   AGE  READY  RESTARTS  GPU  NODE
  ----                                                  ------   ---  -----  --------  ---  ----
  vllm-qwen-4b-chat-v1-custom-serving-6d7c786b9f-z6nfk  Running  36m  1/1    0         1    cn-beijing.192.168.XX.XX

When the Available count matches the Desired count and the instance status is Running, the inference service is ready.

Set up port forwarding

Forward port 8000 from the inference service to your local machine:

kubectl port-forward svc/vllm-qwen-4b-chat-v1 8000:8000

Expected output:

Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000

Send a test request

Open a new terminal and send an inference request to the vLLM inference service:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"  -d '{"model": "/model/Qwen1.5-4B-Chat/", "messages": [{"role": "user", "content": "Test"}], "max_tokens": 10, "temperature": 0.7, "top_p": 0.9, "seed": 10}'

Expected output:

{"id":"cmpl-503270b21fa44db2b6b3c3e0abaa3c02","object":"chat.completion","created":1717141209,"model":"/model/Qwen1.5-4B-Chat/","options":[{"index":0,"message":{"role":"assistant","content":"OK. What do you want to test?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":21,"total_tokens":30,"completion_tokens":9}}

The vLLM inference service exposes an OpenAI-compatible API at the /v1/chat/completions endpoint.