Live subtitles

Usage notes

• The following examples show how to implement this feature using the API without the UI.

• You must integrate AICallKit SDK in advance. For more information, see Integrate AICallKit SDK for Android, Integrate AICallKit SDK for iOS, and Integrate AICallKit SDK for Web.

Overview

How it works

During a call, the onUserSubtitleNotify callback is triggered when the user's question is recognized by the intelligent agent. The onVoiceAgentSubtitleNotify callback is triggered when the intelligent agent generates a response.

onUserSubtitleNotify details

While the user is talking, the agent's recognition result is returned multiple times via notifications. In live subtitle scenarios, the text content can be directly rendered and displayed on the UI. In conversational chat scenarios, the final text content is rendered and displayed on the UI when isSentenceEnd=true is returned.

For example, when a user asks "What's the weather like today?", the following callback results may be returned:

text="What's the weather" isSentenceEnd=false sentenceId=1

text="like" isSentenceEnd=false sentenceId=1

text="today?" isSentenceEnd=true sentenceId=1

onVoiceAgentSubtitleNotify details

The intelligent agent's response is returned in multiple notifications. The client must merge these text fragments and render the combined text incrementally for display on the UI.

For example, in response to the user's question, the agent replied, "Today's weather is clear and sunny, with a mild temperature, so it's a nice day to go outside." The following callback results may be returned:

text="Today's weather is clear and sunny," isSentenceEnd=false userAsrSentenceId=1

text="with a mild temperature, so it's a nice day to go outside." isSentenceEnd=false userAsrSentenceId=1

text="" isSentenceEnd=true userAsrSentenceId=1

Sample code

// Register a callback for a call engine.
mARTCAICallEngine.setEngineCallback(mCallEngineCallbackWrapper); 

// Handle callback events.
ARTCAICallEngine.IARTCAICallEngineCallback mCallEngineCallbackWrapper = new ARTCAICallEngine.IARTCAICallEngineCallback() {
    @Override
    public void onUserAsrSubtitleNotify(String text, boolean isSentenceEnd, int sentenceId, VoicePrintStatusCode voicePrintStatusCode) {
        // Synchronize the ASR-recognized text.
    }

    @Override
    public void onAIAgentSubtitleNotify(String text, boolean end, int userAsrSentenceId) {
        // Synchronize the response of the intelligent agent.
    }
}

// Register a callback for a call engine.
self.engine.delegate = self

func onUserSubtitleNotify(text: String, isSentenceEnd: Bool, sentenceId: Int, voiceprintResult: ARTCAICallVoiceprintResult) {
    // Triggered when the user's question is recognized by the intelligent agent.
}

func onVoiceAgentSubtitleNotify(text: String, isSentenceEnd: Bool, userAsrSentenceId: Int) {
    // Triggered when the intelligent agent generates a response.
}

// Register a callback for a call engine.
engine.on('userSubtitleNotify', (subtitle) => {
  // Triggered when the user's question is recognized by the intelligent agent.
  console.log('AICallUserSubtitleNotify', subtitle.text, subtitle.end, subtitle.sentenceId);
});
engine.on('agentSubtitleNotify', (subtitle) => {
  // Triggered when the intelligent agent generates a response.
  console.log('AICallAgentSubtitleNotify', subtitle.text, subtitle.end, subtitle.sentenceId);
});