Access MLLMs

Updated at: 2025-02-28 09:09

Real-time workflows allow you to access multimodal large language models (MLLMs) based on the specified specifications.

Access self-developed MLLMs that comply with OpenAPI specifications

You can integrate MLLMs that comply with OpenAPI specifications into your workflow based on the OpenAI specifications. You can request MLLMs that comply with OpenAI specifications only in streaming mode.

  1. To integrate an MLLM into a workflow, set the Select Model parameter to Access Self-developed Model (Based on OpenAPI Specifications) and configure the following parameters in the configuration panel of the MLLM node.

    Parameter

    Type

    Required

    Description

    Example

    Parameter

    Type

    Required

    Description

    Example

    ModelId

    String

    Yes

    The model name. This parameter corresponds to the model field in the OpenAI specifications.

    abc

    API-KEY

    String

    Yes

    The authentication information. This parameter corresponds to the api_key field in the OpenAPI specifications.

    AUJH-pfnTNMPBm6iWXcJAcWsrscb5KYaLitQhHBLKrI

    HTTPS URL of Destination Model

    String

    Yes

    The service request URL. This parameter corresponds to the base_url field in the OpenAPI specifications.

    http://www.abc.com

    Maximum Number of Images per Call

    Integer

    Yes

    The maximum number of images in a request to the MLLM. The maximum number of image frames that some MLLMs can receive per request is fixed. You can specify this parameter for these models. When you request an MLLM, frames are extracted from the video for sampling based on the specified value.

    15

  2. When the real-time workflow is running, the OpenAI specifications data is assembled in a POST request and used to access the HTTPS address of the self-developed model that you configure to obtain the corresponding result. The following table describes the input parameters.

    Parameter

    Type

    Description

    Example

    Parameter

    Type

    Description

    Example

    messages

    Array

    The context of historical conversations. A maximum of 20 context records can be retained. A context record at the top of the array indicates an early question or answer.

    Note
    • Only the JPEG Base64-encoded data after frame extraction can be passed.

    • Image data in historical conversations are not delivered as context.

    [
      {
        "role": "user",
        "content": "What is the weather like today?"
      },
      {
        "role": "assistant",
        "content": "It is sunny today."
      },
      {
        "role": "user",
        "content": "What will the weather be like tomorrow?"
      },
      {
        "role": "user",
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/jpeg;base64,xxxx"
            }
          },
          {
            "type": "text",
            "text": "What is this?"
          }
        ]
      }
    ]

    model

    String

    The model name.

    abc

    stream

    Boolean

    Specifies whether to access the model in stream mode. Currently, only the streaming mode is supported.

    True

    extendData

    Object

    The supplementary information.

    {'instanceId':'68e00b6640e*****3e943332fee7','channelId':'123','userData':'{"aaaa":"bbbb"}'}

    • instanceId

    String

    The instance ID.

    68e00b6640e*****3e943332fee7

    • channelId

    String

    The channel ID.

    123

    • userData

    String

    The value of the UserData field that is passed when the instance is started.

    {"aaaa":"bbbb"}

References

Create and manage a workflow template

  • On this page (1, O)
  • Access self-developed MLLMs that comply with OpenAPI specifications
  • References
Feedback
phone Contact Us