This topic describes how to use AICallKit SDK to retrieve live subtitles for both user input and AI agent responses.
Before you begin
The following examples show how to implement this feature using the API without the UI.
You must integrate AICallKit SDK in advance. For more information, see Integrate AICallKit SDK for Android, Integrate AICallKit SDK for iOS, and Integrate AICallKit SDK for Web.
Overview
Live subtitles on the user side The user input is recognized by the AI agent and displayed on the UI in real time.
| Live subtitles on the agent side The content generated by the LLM is displayed on the UI in real time.
|
This feature is included in the integration solution with UI.
How it works
During a call, the onUserSubtitleNotify callback is triggered when the user's question is recognized by the AI agent. The onVoiceAgentSubtitleNotify callback is triggered when the AI agent generates a response.
onUserSubtitleNotify details
While the user is talking, the agent's recognition result is returned multiple times via notifications. In live subtitle scenarios, the text content can be directly rendered and displayed on the UI. In conversational chat scenarios, the final text content is rendered and displayed on the UI when isSentenceEnd=true is returned.
Parameter | Description |
text | The text recognized by the AI agent. |
isSentenceEnd | Indicates whether the current text is the end of the sentence. If this parameter is set to true, the AI agent starts answering the question. |
sentenceId | The ID of the sentence to which the current text belongs. The IDs are accumulated after each question is asked. |
voiceprintResult | The voice activity detection (VAD) result. Valid values: 0: VAD-based noise suppression and AIVad are disabled. 1: VAD-based noise suppression is enabled, but voiceprints are not registered. 2: VAD-based noise suppression is enabled and the main speaker is recognized. 3: VAD-based noise suppression is enabled, but the main speaker is not recognized. In this case, the AI agent will not answer the question. 4: AIVad is enabled and the main speaker is recognized. 5: AIVad is enabled, but the main speaker is not recognized. In this case, the AI agent will not answer the question. Note AICallKit SDK for Web does not support voiceprint-based noise suppression. |
For example, when a user asks "What's the weather like today?", the following callback results may be returned:
text="What's the weather" isSentenceEnd=false sentenceId=1
text="like" isSentenceEnd=false sentenceId=1
text="today?" isSentenceEnd=true sentenceId=1
onVoiceAgentSubtitleNotify details
The AI agent's response is returned in multiple notifications. The client must merge these text fragments and render the combined text incrementally for display on the UI.
Parameter | Description |
text | The text of the answer provided by the AI agent. |
isSentenceEnd | Indicates whether the current text is the final sentence of the answer. If this parameter is set to true, the AI agent finishes answering the question and enters the Listening state. |
userAsrSentenceId | The ID of the sentence that the user's question recognized by the AI agent belongs to. |
For example, in response to the user's question, the agent replied, "Today's weather is clear and sunny, with a mild temperature, so it's a nice day to go outside." The following callback results may be returned:
text="Today's weather is clear and sunny," isSentenceEnd=false userAsrSentenceId=1
text="with a mild temperature, so it's a nice day to go outside." isSentenceEnd=false userAsrSentenceId=1
text="" isSentenceEnd=true userAsrSentenceId=1
Sample code
Android
// Register a callback for a call engine.
mARTCAICallEngine.setEngineCallback(mCallEngineCallbackWrapper);
// Handle callback events.
ARTCAICallEngine.IARTCAICallEngineCallback mCallEngineCallbackWrapper = new ARTCAICallEngine.IARTCAICallEngineCallback() {
@Override
public void onUserAsrSubtitleNotify(String text, boolean isSentenceEnd, int sentenceId, VoicePrintStatusCode voicePrintStatusCode) {
// Synchronize the ASR-recognized text.
}
@Override
public void onAIAgentSubtitleNotify(String text, boolean end, int userAsrSentenceId) {
// Synchronize the response of the AI agent.
}
}iOS
// Register a callback for a call engine.
self.engine.delegate = self
func onUserSubtitleNotify(text: String, isSentenceEnd: Bool, sentenceId: Int, voiceprintResult: ARTCAICallVoiceprintResult) {
// Triggered when the user's question is recognized by the AI agent.
}
func onVoiceAgentSubtitleNotify(text: String, isSentenceEnd: Bool, userAsrSentenceId: Int) {
// Triggered when the AI agent generates a response.
}Web
// Register a callback for a call engine.
engine.on('userSubtitleNotify', (subtitle, voiceprintResult) => {
// Triggered when the user's question is recognized by the AI agent.
console.log('AICallUserSubtitleNotify', subtitle.text, subtitle.end, subtitle.sentenceId, voiceprintResult);
});
engine.on('agentSubtitleNotify', (subtitle) => {
// Triggered when the AI agent generates a response.
console.log('AICallAgentSubtitleNotify', subtitle.text, subtitle.end, subtitle.sentenceId);
});