Integrate into web apps - Intelligent Media Services - Alibaba Cloud Documentation Center

This topic describes how to integrate AICallKit SDK into your web app.

Integrate the SDK

npm install aliyun-auikit-aicall --save

Examples of using the SDK

// Import the SDK.
import ARTCAICallEngine, { ARTCAICallErrorCode, ARTCAICallAgentState, AICallSubtitleData } from 'aliyun-auikit-aicall';

// Create an engine instance.
const engine = new ARTCAICallEngine();

// For more information about how to call other API operations, see the API reference section of this topic.

// The callbacks. The following examples include only callbacks that do not require specialized main code.
engine.on('errorOccurred', (code: ARTCAICallErrorCode) => {
  // An error occurred.
  engine.handup();
});

engine.on('callBegin', () => {
  // A call starts.
});

engine.on('callEnd', () => {
  // A call ends.
});

engine.on('agentStateChanged', (state: ARTCAICallAgentState) => {
  // The status of the intelligent agent changes.
});

engine.on('userSubtitleNotify', (subtitle: AICallSubtitleData) => {
  // A notification is sent when the intelligent agent recognizes the question of the user.
});

engine.on('agentSubtitleNotify', (subtitle: AICallSubtitleData) => {
  // A notification is sent when the intelligent agent answers the question of the user.
});

engine.on('voiceIdChanged', (voiceId: string) => {
  // The voice of the intelligent agent changes.
});

engine.on('voiceInterruptChanged', (enable: boolean) => {
  // Voice interruption is enabled or disabled.
});

const userId = '';

// Obtain the intelligent agent information from the server.
const agentInfo = fetchAgentInfo();

try {
  // Start the call after the intelligent agent is started.
  engine.call(userId, agentInfo);
} catch (error) {}

// End the call.
engine.handup();

API reference

Parameters

Parameter	Description

Parameter	Description
userId	The ID of the user on the current call.
isOnCall	Indicates whether a call is in progress.
agentInfo	The information about the current intelligent agent.
agentState	The status of the current intelligent agent.

API overview

Class or protocol	API	Description

Class or protocol	API	Description
ARTCAICallEngine an engine instance	call	Starts a call.
	handup	Ends a call.
	setAgentView	Configures the rendering view and view mode for the intelligent agent.
	interruptSpeaking	Interrupts the speech of the intelligent agent.
	enableVoiceInterrupt	Enables or disables intelligent interruption.
	startPreview	Sets the URL for local preview.
	muteLocalCamera	Enables or disables camera push.
	switchCamera	Switches between the front and rear cameras.
	switchVoiceId	Changes the voice.
	mute	Mutes or unmutes the microphone.
	startPushToTalk	Starts speaking in intercom mode.
	finishPushToTalk	Finishes speaking in intercom mode.
	cancelPushToTalk	Cancels speaking in intercom mode.
	enablePushToTalk	Enables or disables intercom mode. In intercom mode, the intelligent agent returns a result only after the finishPushToTalk operation is called.
	getRTCInstance	Queries the information about a Real-Time Communication (RTC) engine instance.
	destroy	Releases resources.
ARTCAICallEngine callback events of an engine instance	errorOccurred	An error occurred.
	callBegin	A call starts.
	callEnd	A call ends.
	agentStateChanged	The status of the intelligent agent changes.
	speakingVolumeChanged	The volume changes.
	userSubtitleNotify	The intelligent agent recognizes the question of the user.
	agentSubtitleNotify	The intelligent agent returns an answer.
	voiceIdChanged	The voice of the intelligent agent changes.
	pushToTalkChanged	The intercom mode changes.
	agentWillLeave	The intelligent agent is about to end the current call.
	receivedAgentCustomMessage	A custom message is received from the current intelligent agent.
	voiceInterruptChanged	Indicates whether voice interruption is enabled for the current call.
	humanTakeoverWillStart	A human agent is stepping in to take over from the current intelligent agent.
	humanTakeoverConnected	A human agent has taken over from the current intelligent agent.

ARTCAICallEngine

call

Starts a call.

async call(userId: string, agentInfo: AICallAgentInfo, config?: AICallEngineConfig): Promise<void>>

Parameters

Parameter	Type	Description

Parameter	Type	Description
userId	String	The UID of the current user.
agentInfo	AICallAgentInfo	The information about the intelligent agent.
config	AICallEngineConfig	The initialization configuration. Example: `{ // Specifies whether to mute the microphone. muteMicrophone?: boolean; // Specifies whether to disable the camera in visual understanding mode. muteCamera?: boolean; // Specifies whether to enable intercom mode. enablePushToTalk?: boolean; // The video preview element in visual understanding mode. previewElement?: string \| HTMLVideoElement; // The camera settings in visual understanding mode. cameraConfig?: AICallCameraConfig; }`

handup

Ends a call.

async handup(): Promise<void>

setAgentView

Configures the rendering view and view mode for the intelligent agent.

setAgentView(view: HTMLVideoElement | string): void

Parameters

Parameter	Type	Description

Parameter	Type	Description
view	HTMLVideoElement \| string	The <video> element or its ID.

interruptSpeaking

Interrupts the speech of the intelligent agent.

async interruptSpeaking(): Promise<void>

enableVoiceInterrupt

Enables or disables intelligent interruption.

async enableVoiceInterrupt(enable: boolean): Promise<void>

Parameters

Parameter	Type	Description

Parameter	Type	Description
enable	boolean	Specifies whether to enable intelligent interruption.

startPreview

Sets the URL for local preview.

startPreview(view: HTMLVideoElement | string): void

Parameters

Parameter	Type	Description

Parameter	Type	Description
view	HTMLVideoElement \| string	The <video> element or its ID.

muteLocalCamera

Enables or disables camera push.

async muteLocalCamera(mute: boolean)

Parameters

Parameter	Type	Description

Parameter	Type	Description
mute	boolean	Specifies whether to enable the camera.

switchCamera

Switches between the front and rear cameras.

async switchCamera(deviceId?: string)

Parameters

Parameter	Type	Description

Parameter	Type	Description
deviceId	string	The device ID. You can query the device ID by using ARTCAICallEngine.getCameraList(). If you do not specify this parameter, a camera switch is performed on the mobile phone.

switchVoiceId

Changes the voice.

async switchVoiceId(voiceId: string): Promise<void>

Parameters

Parameter	Type	Description

Parameter	Type	Description
voiceId	string	The voice ID.

mute

Mutes or unmutes the microphone.

async mute(mute: boolean): Promise<void>

Parameters

Parameter	Type	Description

Parameter	Type	Description
mute	boolean	Specifies whether to mute the microphone.

startPushToTalk

Starts speaking in intercom mode.

startPushToTalk() :boolean;

finishPushToTalk

Finishes speaking in intercom mode.

finishPushToTalk() :boolean;

cancelPushToTalk

Cancels speaking in intercom mode.

cancelPushToTalk() :boolean;

enablePushToTalk

Enables or disables intercom mode. In intercom mode, the intelligent agent returns a result only after the finishPushToTalk API operation is called.

enablePushToTalk(enable: boolean):boolean;

Parameters

Parameter	Type	Description

Parameter	Type	Description
enable	Bool	Specifies whether to enable intercom mode.

getRTCInstance

Queries the information about an RTC engine instance.

getRTCInstance(): AliRtcEngine | undefined

destroy

Releases resources.

async destroy()

ARTCAICallEngine

errorOccurred

An error occurred during the current call.

Parameters

Parameter	Type	Description

Parameter	Type	Description
code	AICallErrorCode	The error code.

callBegin

A call starts.

callEnd

A call ends.

agentStateChanged

The status of the intelligent agent changes.

Parameters

Parameter	Type	Description

Parameter	Type	Description
state	AICallAgentState	The status of the intelligent agent. The agent can be listening, thinking, or speaking.

speakingVolumeChanged

The volume changes.

Parameters

Parameter	Type	Description

Parameter	Type	Description
uid	string	The UID of the current speaker. The value is an empty string if the speaker is the current user.
volume	Int32	The volume. Valid values: 0 to 100.

userSubtitleNotify

The intelligent agent recognizes the question of the user.

Parameters

Parameter	Type	Description

Parameter	Type	Description
subtitle	AICallSubtitleData	The subtitle.

agentSubtitleNotify

The intelligent agent returns an answer.

Parameters

Parameter	Type	Description

Parameter	Type	Description
subtitle	AICallSubtitleData	The subtitle.

voiceIdChanged

The voice of the intelligent agent changes.

Parameters

Parameter	Type	Description

Parameter	Type	Description
voiceId	string	The voice ID.

pushToTalkChanged

The intercom mode changes.

Parameters

Parameter	Type	Description

Parameter	Type	Description
enable	boolean	Indicates whether intercom mode is enabled.

agentWillLeave

The intelligent agent is about to end the current call.

Parameters

Parameter	Type	Description

Parameter	Type	Description
reason	number	The reason why the agent is leaving. A value of 2001 indicates idle timeout. A value of 0 indicates other reasons.
message	string	The description of the reason.

receivedAgentCustomMessage

A custom message is received from the current intelligent agent.

Parameters

Parameter	Type	Description

Parameter	Type	Description
data	Object	The message content.

voiceInterruptChanged

Indicates whether voice interruption is enabled for the current call.

Parameters

Parameter	Type	Description

Parameter	Type	Description
enable	boolean	Indicates whether voice interruption is enabled.

humanTakeoverWillStart

A human agent is stepping in to take over from the current intelligent agent.

Parameters

Parameter	Type	Description

Parameter	Type	Description
takeoverUid	string	The UID of the human agent.
takeoverMode	number	1 indicates the use of human voice. 0 indicates the use of intelligent agent voice.

humanTakeoverConnected

A human agent has taken over from the current intelligent agent.

Parameters

Parameter	Type	Description

Parameter	Type	Description
takeoverUid	string	The UID of the human agent.

agentEmotionNotify

The intelligent agent detects an emotion.

Parameters

Parameter	Type	Description

Parameter	Type	Description
emotion	string	The emotion tag, such as neutral, happy, angry, or sad.
userAsrSentenceId	number	The ID of the sentence to which the question of the user recognized by the intelligent agent belongs.