Platform For AI:Use LLM intelligent router to improve inference efficiency

Architecture overview

The LLM Intelligent Router consists of three core components that work together to provide intelligent traffic distribution and management for backend LLM inference instance clusters:

• LLM Gateway: Primary traffic entry point and request processor that supports HTTP/SSE and WebSocket protocols

• LLM Scheduler: Central intelligence that aggregates real-time metrics and makes policy-based scheduling decisions

• LLM Agent: Distributed monitoring probe deployed as sidecar container with each inference instance

The core workflow process:

1. Instance registration: After the inference service starts, the LLM Agent waits for the inference engine to become ready, then registers the instance with the LLM Scheduler and periodically reports health status and performance metrics.

2. Traffic ingestion: User requests are first received by the LLM Gateway.

3. Scheduling decision: The LLM Gateway sends a scheduling request to the LLM Scheduler.

4. Intelligent scheduling: The LLM Scheduler selects the optimal backend instance based on the scheduling policy and real-time metrics reported by all LLM Agents.

5. Request forwarding: The LLM Scheduler returns the decision to the LLM Gateway, which forwards the user's original request to the target instance.

6. Request buffering: If all backend instances are under high load, new requests are temporarily buffered in the LLM Gateway's queue, waiting for the LLM Scheduler to find a suitable forwarding opportunity to prevent request failures.

Core components

Failover mechanism

The system implements multiple layers of fault tolerance mechanisms to ensure service stability:

• LLM Gateway (high availability): The LLM Gateway is a stateless traffic ingress layer. For high availability, you should deploy at least two instances. If one instance fails, traffic automatically switches to the healthy instances, which ensures continuous service availability.

• LLM Scheduler (degraded fault tolerance): The LLM Scheduler is the request scheduling component and runs as a single instance to enable global scheduling. If the LLM Scheduler fails, the LLM Gateway automatically enters a degraded mode after a heartbeat failure. In this mode, it uses a round-robin policy to forward requests directly to backend instances. This ensures service availability but sacrifices scheduling performance. After the LLM Scheduler recovers, the LLM Gateway automatically resumes the intelligent scheduling mode.

• Inference instance or LLM Agent (automatic removal): If an inference instance or its associated LLM Agent fails, the heartbeat between the LLM Agent and the LLM Scheduler is interrupted. The LLM Scheduler then immediately removes the instance from the list of available instances and stops assigning new traffic to it. After the instance recovers and re-establishes its heartbeat, it is automatically added back to the service list.

Multi-inference engine support

Because different LLM inference engines return different metric information through their /metrics endpoints, the LLM Agent collects these metrics and standardizes their format before reporting them. This means the LLM Scheduler does not need to understand the implementation details of a specific inference engine. It only needs to implement scheduling algorithms based on the standardized metrics. The currently supported LLM inference engines and their corresponding collected metrics are as follows:

Step 1: Deploy LLM intelligent router service

1. Log on to the PAI console and select the destination region at the top of the page.

2. In the navigation pane on the left, click Elastic Algorithm Service (EAS), select the target workspace, and go to the EAS page.

3. Click Deploy Service, and then choose Scenario-based Model Deployment > Deploy LLM gateway.

4. Configure the parameters:

   Parameter		Description
   Basic Information	Service Name	Customize a service name, such as llm_gateway.
   Resource Information	Deployment Resources	Resource configuration for LLM Gateway. To ensure high availability, the default Number of Replicas is 2. Keep this setting. Default CPU is 4 cores and memory is 8 GB.
   	Scheduling configuration	Resource configuration for LLM Scheduler. Default CPU is 2 cores and memory is 4 GB.
   	Scheduling Policy	Select the load balancing policy for backend inference instances. Default is Prefix cache. For detailed comparisons and selection guidance, see Scheduling Policy Details and Selection.

5. Click Deploy. When the service status changes to Running, the deployment is successful.

After a successful deployment, the system automatically creates a service group named group_<LLM intelligent router service name>. You can go to the Elastic Algorithm Service (EAS) page and view the service group on the Canary Release tab.

Step 2: Deploy an LLM service

You must configure the LLM intelligent router feature when you deploy a new LLM service. You cannot add this feature by updating an existing LLM service.

The following steps show how to deploy Qwen3-8B:

1. You can click Deploy Service and select Scenario-based Model Deployment > LLM Deployment.

2. You can configure the following key parameters:

   Parameter		Value
   Basic Information	Model Settings	Select Public Model. Then, search for and select Qwen3-8B.
   	Inference Engine	Select vLLM (Recommended, compatible with the OpenAI API). Note If you select the Prefix cache scheduling policy for the LLM intelligent router service, you must enable the prefix cache feature when you deploy an LLM service that uses vLLM as the inference engine.
   	Deployment Template	Select Standalone. The system automatically fills in recommended parameters, such as instance type and runtime image, based on the template.
   Features	LLM Intelligent Router	Turn on the switch and select the LLM intelligent router service deployed in Step 1 from the drop-down list.

3. You can click Deploy. The service deployment takes about 5 minutes. When the service status changes to Running, the deployment is successful.

Step 3: Test the service

You must send all requests to the LLM intelligent router service endpoint, not to the endpoints of specific backend inference services.

1. Obtain access credentials.

   1. Click the LLM intelligent router service to go to the Overview page. In the Basic Information section, click View Endpoint Information.

   2. On the Endpoint Information page, copy the Internet Endpoint and Token from the Service-specific Traffic Entry section.

2. Construct the request URL and call the service.

   • URL structure: <LLM intelligent router endpoint>/<LLM service API path>

   • Example: http://********.pai-eas.aliyuncs.com/api/predict/group_llm_gateway.llm_gateway/v1/chat/completions

   Request example:

   # Replace <YOUR_GATEWAY_URL> and <YOUR_TOKEN> with your actual values
   # Replace <model_name> with your actual model name
   curl -X POST "<YOUR_GATEWAY_URL>/v1/chat/completions" \
        -H "Authorization: Bearer <YOUR_TOKEN>" \
        -H "Content-Type: application/json" \
        -N \
        -d '{
              "model": "<model_name>",
              "messages": [{"role": "user", "content": "Hello"}],
              "stream": true
            }'

   Sample response:

   data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}]}
   data: {"id":"chatcmpl-9a9f8299*****","object":"chat.completion.chunk","created":1762245102,"model":"Qwen3-8B","choices":[{"index":0,"delta":{"content":"<think>","tool_calls":[]}}]}
   
   ...
   data: [DONE]