Access MLLMs

Access self-developed MLLMs that comply with OpenAPI specifications

You can integrate MLLMs that comply with OpenAPI specifications into your workflow based on the OpenAI specifications. You can request MLLMs that comply with OpenAI specifications only in streaming mode.

1. To integrate an MLLM into a workflow, set the Select Model parameter to Access Self-developed Model (Based on OpenAPI Specifications) and configure the following parameters in the configuration panel of the MLLM node.

   Parameter	Type	Required	Description	Example

   Parameter	Type	Required	Description	Example
   ModelId	String	Yes	The model name. This parameter corresponds to the model field in the OpenAI specifications.	abc
   API-KEY	String	Yes	The authentication information. This parameter corresponds to the api_key field in the OpenAPI specifications.	AUJH-pfnTNMPBm6iWXcJAcWsrscb5KYaLitQhHBLKrI
   HTTPS URL of Destination Model	String	Yes	The service request URL. This parameter corresponds to the base_url field in the OpenAPI specifications.	http://www.abc.com
   Maximum Number of Images per Call	Integer	Yes	The maximum number of images in a request to the MLLM. The maximum number of image frames that some MLLMs can receive per request is fixed. You can specify this parameter for these models. When you request an MLLM, frames are extracted from the video for sampling based on the specified value.	15

2. When the real-time workflow is running, the OpenAI specifications data is assembled in a POST request and used to access the HTTPS address of the self-developed model that you configure to obtain the corresponding result. The following table describes the input parameters.

   Parameter	Type	Description	Example

   Parameter	Type	Description	Example
   messages	Array	The context of historical conversations. A maximum of 20 context records can be retained. A context record at the top of the array indicates an early question or answer. Note Only the JPEG Base64-encoded data after frame extraction can be passed. Image data in historical conversations are not delivered as context.	[ { "role": "user", "content": "What is the weather like today?" }, { "role": "assistant", "content": "It is sunny today." }, { "role": "user", "content": "What will the weather be like tomorrow?" }, { "role": "user", "content": [ { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,xxxx" } }, { "type": "text", "text": "What is this?" } ] } ]
   model	String	The model name.	abc
   stream	Boolean	Specifies whether to access the model in stream mode. Currently, only the streaming mode is supported.	True
   extendData	Object	The supplementary information.	{'instanceId':'68e00b6640e*****3e943332fee7','channelId':'123','userData':'{"aaaa":"bbbb"}'}
   instanceId	String	The instance ID.	68e00b6640e*****3e943332fee7
   channelId	String	The channel ID.	123
   userData	String	The value of the UserData field that is passed when the instance is started.	{"aaaa":"bbbb"}