Spatial streaming from iPhone

Hi, I am trying to stream spatial video in realtime from my iPhone 16.

I am able to record spatial video as a file output using:

let videoDeviceOutput = AVCaptureMovieFileOutput()

However, when I try to grab the raw sample buffer, it doesn't include any spatial information:

let captureOutput = AVCaptureVideoDataOutput()

//when init camera
session.addOutput(captureOutput)
captureOutput.setSampleBufferDelegate(self, queue: sessionQueue)

//finally
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
       //use sample buffer (but no spatial data available here)
}

Is this how it's supposed to work or maybe I am missing something?

this video: https://developer.apple.com/videos/play/wwdc2023/10071 gives us a clue towards setting up spatial streaming and I've got the backend all ready for 3D HLS streaming. Now I am only stuck at how to send the video stream to my server.

Hello @AbhiStack

Spatial data is not available via AVCaptureVideoDataOutput. I asked around and streaming spatial video from your iPhone 16 should be possible, but you have to do manual work. Here's some information to point you in the right direction.

Spatial streaming from iPhone
 
 
Q