Platform For AI：5分鐘使用EAS一鍵部署LLM大語言模型應用

使用HTTP方式調用服務

• 非流式調用

  用戶端使用標準的HTTP格式，使用命令列調用時，支援發送以下兩種類型的請求：

  • 發送String類型的請求

    curl $host -H 'Authorization: $authorization' --data-binary @chatllm_data.txt -v

    其中：$authorization需替換為服務Token；$host：需替換為服務訪問地址；chatllm_data.txt：該檔案為包含問題的純文字檔案，例如加拿大的首都是哪裡？。

  • 發送結構化類型的請求

    curl $host -H 'Authorization: $authorization' -H "Content-type: application/json" --data-binary @chatllm_data.json -v -H "Connection: close"

    使用chatllm_data.json檔案來設定推理參數，chatllm_data.json檔案的內容格式如下：

    {
      "max_new_tokens": 4096,
      "use_stream_chat": false,
      "prompt": "What is the capital of Canada?",
      "system_prompt": "Act like you are a knowledgeable assistant who can provide information on geography and related topics.",
      "history": [
        [
          "Can you tell me what's the capital of France?",
          "The capital of France is Paris."
        ]
      ],
      "temperature": 0.8,
      "top_k": 10,
      "top_p": 0.8,
      "do_sample": true,
      "use_cache": true
    }

    參數說明如下，請酌情添加或刪除。

    參數	描述	預設值
    max_new_tokens	產生輸出token的最大長度，單位為個。	2048
    use_stream_chat	是否使用流式輸出形式。	true
    prompt	使用者的Prompt。	""
    system_prompt	系統Prompt。	""
    history	對話的記錄，類型為List[Tuple(str, str)]。	[()]
    temperature	用於調節模型輸出結果的隨機性，值越大隨機性越強，0值為固定輸出。Float類型，區間為0~1。	0.95
    top_k	從產生結果中選擇候選輸出的數量。	30
    top_p	從產生結果中按百分比選擇輸出結果。Float類型，區間為0~1。	0.8
    do_sample	開啟輸出採樣。	true
    use_cache	開啟KV Cache。	true

  • 您可以使用Python的requests庫來構建自己的用戶端，範例程式碼如下。您可以通過命令列參數--prompt來指定請求的內容，例如：python xxx.py --prompt "What is the capital of Canada?"。

    import argparse
    import json
    from typing import Iterable, List
    
    import requests
    
    def post_http_request(prompt: str,
                          system_prompt: str,
                          history: list,
                          host: str,
                          authorization: str,
                          max_new_tokens: int = 2048,
                          temperature: float = 0.95,
                          top_k: int = 1,
                          top_p: float = 0.8,
                          langchain: bool = False,
                          use_stream_chat: bool = False) -> requests.Response:
        headers = {
            "User-Agent": "Test Client",
            "Authorization": f"{authorization}"
        }
        if not history:
            history = [
                (
                    "San Francisco is a",
                    "city located in the state of California in the United States. \
                    It is known for its iconic landmarks, such as the Golden Gate Bridge \
                    and Alcatraz Island, as well as its vibrant culture, diverse population, \
                    and tech industry. The city is also home to many famous companies and \
                    startups, including Google, Apple, and Twitter."
                )
            ]
        pload = {
            "prompt": prompt,
            "system_prompt": system_prompt,
            "top_k": top_k,
            "top_p": top_p,
            "temperature": temperature,
            "max_new_tokens": max_new_tokens,
            "use_stream_chat": use_stream_chat,
            "history": history
        }
        if langchain:
            pload["langchain"] = langchain
        response = requests.post(host, headers=headers,
                                 json=pload, stream=use_stream_chat)
        return response
    
    def get_response(response: requests.Response) -> List[str]:
        data = json.loads(response.content)
        output = data["response"]
        history = data["history"]
        return output, history
    
    if __name__ == "__main__":
        parser = argparse.ArgumentParser()
        parser.add_argument("--top-k", type=int, default=4)
        parser.add_argument("--top-p", type=float, default=0.8)
        parser.add_argument("--max-new-tokens", type=int, default=2048)
        parser.add_argument("--temperature", type=float, default=0.95)
        parser.add_argument("--prompt", type=str, default="How can I get there?")
        parser.add_argument("--langchain", action="store_true")
    
        args = parser.parse_args()
    
        prompt = args.prompt
        top_k = args.top_k
        top_p = args.top_p
        use_stream_chat = False
        temperature = args.temperature
        langchain = args.langchain
        max_new_tokens = args.max_new_tokens
    
        host = "EAS服務公網地址"
        authorization = "EAS服務公網Token"
    
        print(f"Prompt: {prompt!r}\n", flush=True)
        # 在用戶端請求中可設定語言模型的system prompt。
        system_prompt = "Act like you are programmer with \
                    5+ years of experience."
    
        # 用戶端請求中可設定對話的歷史資訊，用戶端維護目前使用者的對話記錄，用於實現多輪對話。通常情況下可以使用上一輪對話返回的histroy資訊，history格式為List[Tuple(str, str)]。
        history = []
        response = post_http_request(
            prompt, system_prompt, history,
            host, authorization,
            max_new_tokens, temperature, top_k, top_p,
            langchain=langchain, use_stream_chat=use_stream_chat)
        output, history = get_response(response)
        print(f" --- output: {output} \n --- history: {history}", flush=True)
    
    # 服務端返回JSON格式的響應結果，包含推理結果與對話歷史。
    def get_response(response: requests.Response) -> List[str]:
        data = json.loads(response.content)
        output = data["response"]
        history = data["history"]
        return output, history

    其中：

    • host：配置為服務訪問地址。

    • authorization：配置為服務Token。

• 流式調用

  流式調用使用HTTP SSE方式，其他設定與非流式相同，代碼參考如下。您可以通過命令列參數--prompt來指定請求的內容，例如python xxx.py --prompt "What is the capital of Canada?"。

  import argparse
  import json
  from typing import Iterable, List
  
  import requests
  
  
  def clear_line(n: int = 1) -> None:
      LINE_UP = '\033[1A'
      LINE_CLEAR = '\x1b[2K'
      for _ in range(n):
          print(LINE_UP, end=LINE_CLEAR, flush=True)
  
  
  def post_http_request(prompt: str,
                        system_prompt: str,
                        history: list,
                        host: str,
                        authorization: str,
                        max_new_tokens: int = 2048,
                        temperature: float = 0.95,
                        top_k: int = 1,
                        top_p: float = 0.8,
                        langchain: bool = False,
                        use_stream_chat: bool = False) -> requests.Response:
      headers = {
          "User-Agent": "Test Client",
          "Authorization": f"{authorization}"
      }
      if not history:
          history = [
              (
                  "San Francisco is a",
                  "city located in the state of California in the United States. \
                  It is known for its iconic landmarks, such as the Golden Gate Bridge \
                  and Alcatraz Island, as well as its vibrant culture, diverse population, \
                  and tech industry. The city is also home to many famous companies and \
                  startups, including Google, Apple, and Twitter."
              )
          ]
      pload = {
          "prompt": prompt,
          "system_prompt": system_prompt,
          "top_k": top_k,
          "top_p": top_p,
          "temperature": temperature,
          "max_new_tokens": max_new_tokens,
          "use_stream_chat": use_stream_chat,
          "history": history
      }
      if langchain:
          pload["langchain"] = langchain
      response = requests.post(host, headers=headers,
                               json=pload, stream=use_stream_chat)
      return response
  
  
  def get_streaming_response(response: requests.Response) -> Iterable[List[str]]:
      for chunk in response.iter_lines(chunk_size=8192,
                                       decode_unicode=False,
                                       delimiter=b"\0"):
          if chunk:
              data = json.loads(chunk.decode("utf-8"))
              output = data["response"]
              history = data["history"]
              yield output, history
  
  
  if __name__ == "__main__":
      parser = argparse.ArgumentParser()
      parser.add_argument("--top-k", type=int, default=4)
      parser.add_argument("--top-p", type=float, default=0.8)
      parser.add_argument("--max-new-tokens", type=int, default=2048)
      parser.add_argument("--temperature", type=float, default=0.95)
      parser.add_argument("--prompt", type=str, default="How can I get there?")
      parser.add_argument("--langchain", action="store_true")
      args = parser.parse_args()
  
      prompt = args.prompt
      top_k = args.top_k
      top_p = args.top_p
      use_stream_chat = True
      temperature = args.temperature
      langchain = args.langchain
      max_new_tokens = args.max_new_tokens
  
      host = ""
      authorization = ""
  
      print(f"Prompt: {prompt!r}\n", flush=True)
      system_prompt = "Act like you are programmer with \
                  5+ years of experience."
      history = []
      response = post_http_request(
          prompt, system_prompt, history,
          host, authorization,
          max_new_tokens, temperature, top_k, top_p,
          langchain=langchain, use_stream_chat=use_stream_chat)
  
      for h, history in get_streaming_response(response):
          print(
              f" --- stream line: {h} \n --- history: {history}", flush=True)
  

  其中：

  • host：配置為服務訪問地址。

  • authorization：配置為服務Token。

使用WebSocket方式調用服務

為了更好地維護使用者對話資訊，您也可以使用WebSocket方式保持與服務的串連完成單輪或多輪對話，程式碼範例如下：

import os
import time
import json
import struct
from multiprocessing import Process

import websocket

round = 5
questions = 0


def on_message_1(ws, message):
    if message == "<EOS>":
        print('pid-{} timestamp-({}) receives end message: {}'.format(os.getpid(),
              time.time(), message), flush=True)
        ws.send(struct.pack('!H', 1000), websocket.ABNF.OPCODE_CLOSE)
    else:
        print("{}".format(time.time()))
        print('pid-{} timestamp-({}) --- message received: {}'.format(os.getpid(),
              time.time(), message), flush=True)


def on_message_2(ws, message):
    global questions
    print('pid-{} --- message received: {}'.format(os.getpid(), message))
    # end the client-side streaming
    if message == "<EOS>":
        questions = questions + 1
        if questions == 5:
            ws.send(struct.pack('!H', 1000), websocket.ABNF.OPCODE_CLOSE)


def on_message_3(ws, message):
    print('pid-{} --- message received: {}'.format(os.getpid(), message))
    # end the client-side streaming
    ws.send(struct.pack('!H', 1000), websocket.ABNF.OPCODE_CLOSE)


def on_error(ws, error):
    print('error happened: ', str(error))


def on_close(ws, a, b):
    print("### closed ###", a, b)


def on_pong(ws, pong):
    print('pong:', pong)

# stream chat validation test
def on_open_1(ws):
    print('Opening Websocket connection to the server ... ')
    params_dict = {}
    params_dict['prompt'] = """Show me a golang code example: """
    params_dict['temperature'] = 0.9
    params_dict['top_p'] = 0.1
    params_dict['top_k'] = 30
    params_dict['max_new_tokens'] = 2048
    params_dict['do_sample'] = True
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    # raw_req = f"""To open a Websocket connection to the server: """

    ws.send(raw_req)
    # end the client-side streaming


# multi-round query validation test
def on_open_2(ws):
    global round
    print('Opening Websocket connection to the server ... ')
    params_dict = {"max_new_tokens": 6144}
    params_dict['temperature'] = 0.9
    params_dict['top_p'] = 0.1
    params_dict['top_k'] = 30
    params_dict['use_stream_chat'] = True
    params_dict['prompt'] = "您好！"
    params_dict = {
        "system_prompt":
        "Act like you are programmer with 5+ years of experience."
    }
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    ws.send(raw_req)
    params_dict['prompt'] = "請使用Python，編寫一個排序演算法"
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    ws.send(raw_req)
    params_dict['prompt'] = "請轉寫成java語言的實現"
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    ws.send(raw_req)
    params_dict['prompt'] = "請介紹一下你自己？"
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    ws.send(raw_req)
    params_dict['prompt'] = "請總結上述對話"
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    ws.send(raw_req)


# Langchain validation test.
def on_open_3(ws):
    global round
    print('Opening Websocket connection to the server ... ')

    params_dict = {}
    # params_dict['prompt'] = """To open a Websocket connection to the server: """
    params_dict['prompt'] = """Can you tell me what's the MNN?"""
    params_dict['temperature'] = 0.9
    params_dict['top_p'] = 0.1
    params_dict['top_k'] = 30
    params_dict['max_new_tokens'] = 2048
    params_dict['use_stream_chat'] = False
    params_dict['langchain'] = True
    raw_req = json.dumps(params_dict, ensure_ascii=False).encode('utf8')
    ws.send(raw_req)


authorization = ""
host = "ws://" + ""


def single_call(on_open_func, on_message_func, on_clonse_func=on_close):
    ws = websocket.WebSocketApp(
        host,
        on_open=on_open_func,
        on_message=on_message_func,
        on_error=on_error,
        on_pong=on_pong,
        on_close=on_clonse_func,
        header=[
            'Authorization: ' + authorization],
    )

    # setup ping interval to keep long connection.
    ws.run_forever(ping_interval=2)


if __name__ == "__main__":
    for i in range(5):
        p1 = Process(target=single_call, args=(on_open_1, on_message_1))
        p2 = Process(target=single_call, args=(on_open_2, on_message_2))
        p3 = Process(target=single_call, args=(on_open_3, on_message_3))

        p1.start()
        p2.start()
        p3.start()

        p1.join()
        p2.join()
        p3.join()

其中：

• authorization：配置為服務Token。

• host：配置為服務訪問地址。並將訪問地址中前端的http替換為ws。

• use_stream_chat：通過該請求參數來控制用戶端是否為流式輸出。預設值為True，表示服務端返迴流式資料。

• 參考上述範例程式碼中的on_open_2函數的實現方法實現多輪對話。

Python

from openai import OpenAI

##### API 配置 #####
openai_api_key = "<EAS API KEY>"
openai_api_base = "<EAS API Endpoint>/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id
print(model)


def main():

    stream = True

    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "加拿大的首都在哪裡？",
                    }
                ],
            }
        ],
        model=model,
        max_completion_tokens=2048,
        stream=stream,
    )

    if stream:
        for chunk in chat_completion:
            print(chunk.choices[0].delta.content, end="")
    else:
        result = chat_completion.choices[0].message.content
        print(result)


if __name__ == "__main__":
    main()

其中：

• <EAS API KEY>：替換為已查詢的服務Token。

• <EAS API Endpoint>：替換為已查詢的服務訪問地址。

命令列

curl -X POST <service_url>/v1/chat/completions -d '{
    "model": "Qwen2.5-7B-Instruct",
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a helpful and harmless assistant."
                }
            ]
        },
        {
            "role": "user",
            "content": "加拿大的首都在哪裡?"
        }
    ]
}' -H "Content-Type: application/json" -H "Authorization: <your-token>"

其中：

• <service_url>：替換為已查詢的服務訪問地址。

• <your-token>：替換為已查詢的服務Token。