Raw video with alpha channel

Session hosts who hold a raw streaming token can enable alpha channel mode, which requests that Zoom's Video SDK send both the original video data and an alpha mask marking the edge of a user's body. The Video SDK can then use the alpha mask and raw data to remove the user's background and render session users natively in the host's virtual world.

Turn on alpha channel mode

Before turning on the alpha channel, call the following:

• canEnableAlphaChannelMode to check whether the alpha channel mode can be enabled in the current session state.
• isDeviceSupportAlphaChannelMode to check whether the device hardware is capable of supporting video alpha mode.

To turn the alpha channel on or off, call enableAlphaChannelMode.

if let videoHelper = ZoomVideoSDK.shareInstance()?.getVideoHelper(), videoHelper.canEnableAlphaChannelMode() {
    videoHelper.isDeviceSupportAlphaChannelMode() // Determines whether the device hardware is capable of supporting video alpha mode.
    videoHelper.enableAlphaChannelMode(true) // true/false to enable/disable alpha channel.
    videoHelper.isAlphaChannelModeEnabled()
}

ZoomVideoSDKVideoHelper *videoHelper = [[ZoomVideoSDK shareInstance] getVideoHelper];
if (videoHelper != nil) {
    if (videoHelper.canEnableAlphaChannelMode) {
        videoHelper.isDeviceSupportAlphaChannelMode; // Determines whether the device hardware is capable of supporting video alpha mode.
        [videoHelper enableAlphaChannelMode:TRUE]; // TRUE/FALSE to enable/disable alpha channel.
        videoHelper.isAlphaChannelModeEnabled;
    }
}

Alpha channel callbacks

The onVideoAlphaChannelStatusChanged callback is triggered when alpha channel mode is turned on or off. After turning on the alpha channel, the raw video data includes two more values, alphaBuffer for the alpha data buffer and alphaBufferLen for the alpha data buffer length, in the same onRawDataFrameReceived callback.

Rely on onVideoAlphaChannelStatusChanged to know when alpha data is available, rather than null-checking alphaBuffer on every frame. Once you've received the data, edit the raw data to suit your use case. The alpha plane marks the user's silhouette against the Y, U, and V color planes, so read it alongside them when compositing.

func onVideoAlphaChannelStatusChanged(_ isAlphaChannelOn: Bool) {
}
func onRawDataFrameReceived(_ rawData: ZoomVideoSDKVideoRawData?) {
    guard let rawData = rawData else { return }
    // The Y, U, and V planes hold the color video.
    rawData.yBuffer
    rawData.uBuffer
    rawData.vBuffer
    // The alpha plane marks the edge of the user's body.
    rawData.alphaBuffer // Contains the buffer address only if alpha data exists. Otherwise null.
    rawData.alphaBufferLen
}

- (void)onVideoAlphaChannelStatusChanged:(BOOL)isAlphaChannelOn {
}
- (void)onRawDataFrameReceived:(ZoomVideoSDKVideoRawData *)rawData {
    if (!rawData) {
        return;
    }
    // The Y, U, and V planes hold the color video.
    rawData.yBuffer;
    rawData.uBuffer;
    rawData.vBuffer;
    // The alpha plane marks the edge of the user's body.
    rawData.alphaBuffer; // Contains the buffer address only if alpha data exists. Otherwise null.
    rawData.alphaBufferLen;
}