Whilst AVCaptureSession is setup to capture ProRes RAW video, is it possible to get video pixel data which can read and processed, such as using CIImage(cvPixelBuffer: )
AVCaptureVideoDataOutput outputs ProRes RAW in kCVPixelFormatType_96VersatileBayerPacked12 pixel format. Is there a provided way to debayer this pixel format into something more usable?
I’ve been working on the same problem on an iPhone 17 Pro Max. There doesn’t appear to be a one-call Core Image debayer path for the kCVPixelFormatType_96VersatileBayerPacked12 buffers delivered by AVCaptureVideoDataOutput, but I was able to process them directly with Metal.
On the Main camera’s 4224×3024 btp2 format I receive 6336 bytes/row, which is exactly 12 bits/sensel. The buffer on this device decodes correctly using a MIPI RAW12-style 2-pixels/3-bytes layout. The sample buffers also contain ProRes RAW attachments for Bayer pattern, black/white level, WB multipliers, gain and the Camera RGB→XYZ D65 color matrix.
The pipeline I currently have working is:
unpack 12-bit → normalize black/white → apply WB to Bayer sensels → debayer → Camera RGB→XYZ D65 matrix → gain → display transform
I’m using CVMetalBufferCacheCreateBufferFromImage to access the Core Video backing memory directly from Metal rather than copying the frame through the CPU.
One caveat: I would not assume every btp2 buffer uses the exact same packing. The MIPI-style layout is what I’ve validated on this specific iPhone 17 Pro Max Main-camera format.
If useful I can post the unpack kernel / metadata extraction details.
