Raw data overview

Raw data access gives your app a hook into the Video SDK for iOS's media pipeline: the YUV video frames, PCM audio samples, and screen-share frames the SDK is sending and receiving.

  • Custom video effects: background removal, filters, watermarks, or AR overlays applied per-frame before frames reach remote users.
  • Server-side processing: transcription, recording to disk, or piping audio into your own backend.
  • Custom sources: feed video from a hardware capture device, a media file, or a synthesized stream into the SDK as if it were the device camera or microphone.

The Video SDK provides you with an option to access real-time raw audio, video, and share data during a session, so you can apply additional effects and enhance the user experience. Use a processor component to apply effects or transformations to each piece of data in real-time.

An example of raw share data is when a user shares their screen.

  • Receive and send raw share video data using the same logic as video data, except get the share pipe instead of the video pipe in the raw data pipe method.
  • Receive and send raw share audio data using the same method used for raw audio data.

Follow the process outlined in this flowchart to capture and process raw data.

Flowchart outlining the raw data flow

Select raw data memory mode

In order to utilize raw data of any type, you must first select the memory mode you wish to use. The SDK supports heap-based and stack-based memory modes.

Stack-based memory

  • Variables are allocated automatically and deallocated after the data leaves scope.
  • Variables are not accessible from or transferable to other threads.
  • Typically has faster access than heap-based memory allocation.
  • Memory space is managed by the CPU and will not become fragmented.
  • Variables cannot be resized.

See Stack-based memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryMode.ZoomVideoSDKRawDataMemoryModeStack

Heap-based memory

  • Variables are allocated and deallocated manually. You must allocate and free variables to avoid memory leaks.
  • Variables can be accessed globally.
  • Has relatively slower access than stack-based memory allocation.
  • Has no guarantee on the efficiency of memory space and can become fragmented.
  • Variables can be resized.

See Heap-based dynamic memory allocation for details.

Specify with:

ZoomVideoSDKRawDataMemoryMode.ZoomVideoSDKRawDataMemoryModeHeap

Specify memory mode

After determining which memory mode is right for you, you must specify it when the SDK is initialized. This must be done for audio, video, and share individually. To specify a raw data memory mode, provide one of these enum cases to the ZoomVideoSDKInitParams during SDK initialization.

let initParams = ZoomVideoSDKInitParams()
// Set audio memory mode to heap.
initParams.audioRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap
// Set video memory mode to heap.
initParams.videoRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap
// Set share memory mode to heap.
initParams.shareRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap

ZoomVideoSDKInitParams *initParams = [[ZoomVideoSDKInitParams alloc] init];
// Set audio memory mode to heap.
initParams.audioRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
// Set video memory mode to heap.
initParams.videoRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;
// Set share memory mode to heap.
initParams.shareRawdataMemoryMode = ZoomVideoSDKRawDataMemoryModeHeap;

Once your memory mode is set, you can receive raw data from other users and send raw data from your own custom sources.