ApsaraVideo Live:Switch between layouts based on SEI

When the layout changes from single-streamer streaming to co-streaming, ApsaraVideo Live automatically adds a supplemental enhancement information (SEI) frame to the stream. The SEI frame contains layout information about all participants in the live stream. You can enable SEI listening on the viewer clients to allow them to detect layout changes. This helps address the latency issue when the stream is pulled from a content delivery network (CDN). You can configure SEI settings in the ApsaraVideo Live console.

The following table describes the structure of an SEI frame.

If a single streamer is streaming, the SEI message that viewers receive contains information about only one participant. If co-streaming or battle occurs, the SEI message that viewers receive contains information about multiple participants.

For example, when the streamer whose user ID is 111 is streaming, an SEI frame in the following format is sent to the viewer side:

{"stream":[{"uid":"111","paneid":-1,"zorder":0,"x":0,"y":0,"w":0,"h":0,"type":0,"ms":0,"vol":0,"vad":0}]}

When the streamer whose user ID is 111 is co-streaming with a co-streamer whose user ID is 222, an SEI frame in the following format is sent to the viewer side:

{"stream":[{"uid":"111","paneid":0,"zorder":1,"x":0,"y":0.25,"w":0.5,"h":0.5,"type":0,"ms":0,"vol":1,"vad":119},{"uid":"222","paneid":1,"zorder":1,"x":0.5018382,"y":0.25,"w":0.5,"h":0.5,"type":0,"ms":0,"vol":60,"vad":123}]}

Viewers can determine whether a streaming layout changes based on the number of stream arrays. If only one stream array exists, a single streamer is streaming. If more than one stream array exists, co-streaming or battle occurs. The layout information of the participants indicates the position of each participant in the mixed stream.

The following sample code provides an example on how to use ApsaraVideo Player to parse an SEI frame:

Sample code for Android:

mAliPlayer = AliPlayerFactory.createAliPlayer(mContext);

PlayerConfig playerConfig = mAliPlayer.getConfig();
// Configure SEI frame parsing for audio-only or video-only streams in the FLV format to reduce the startup time.
// Configure the start buffer time. A larger value indicates more stable startup. However, an excessively large value may affect the startup time. Specify this parameter based on your business requirements.
playerConfig.mStartBufferDuration = 1000;
// Configure the cache required for stuttering to recover. Specify a larger value in poor network conditions. We recommend that you set the value to 500 for an audio-only stream and 3000 for a video stream.
playerConfig.mHighBufferDuration = 500;
// Enable SEI listening.
playerConfig.mEnableSEI = true;
mAliPlayer.setConfig(playerConfig);

mAliPlayer.setAutoPlay(true);

mAliPlayer.setOnErrorListener(errorInfo -> {
    mAliPlayer.prepare();
});

mAliPlayer.setOnSeiDataListener(new IPlayer.OnSeiDataListener() {
    @Override
    public void onSeiData(int i, byte[] bytes) {

    }
});

Sample code for iOS:

self.cdnPlayer = [[AliPlayer alloc] init];
self.cdnPlayer.delegate = self;
AVPConfig *config = [self.cdnPlayer getConfig];
config.enableSEI = YES;
[self.cdnPlayer setConfig:config];

// Listen to SEI-related callbacks.
- (void)onSEIData:(AliPlayer*)player type:(int)type data:(NSData *)data {
    if (data.bytes){
        NSString *str = [NSString stringWithUTF8String:data.bytes];
        // Process the SEI message.
    }
}