Processing AVCaptureVideoDataOutput video stream with appleLog and HLG_BT2020 AVCaptureColorSpace input

Question

Created 1w

Replies 1

Boosts 0

Participants 2

I’m building a professional camera app where users can customize the video recording format and color grading. In the func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) method, I handle video frames and use Metal for real-time color grading. This works well when device.activeColorSpace is sRGB or P3, and the results are great. However, when the color space is HLG_BT2020 or appleLog, the MTKTextureLoader.newTexture(cgImage: cgImage, options: options) method throws an error. After researching, I found that the video frame in these color spaces has a bit-per-channel (bpc) greater than 8 after being converted to CGImage, causing the texture creation to fail. I tried converting the CGImage to a lower bpc to successfully create the texture, but the final output image is garbled and not as expected. Is there a solution to this issue?

Boost

Answer 1

MikeAlpha OP

5d

Hello!

I don't have an experience in this particular situation. However, I spent a lot of time working with GPUs and video (either from live camera or from decoder). So this is just a guess and please take it with a grain of salt:

Apple Metal does support textures with more than 8 bits per channel, for example MTLPixelFormatRGB10A2Unorm or MTLPixelFormatBGRA10_XR.
However, I don't think that Metal supports typical video (YUV) formats with more than 8 bits per channel. Like if you had, say, h.265 stream using Main10 profile with 10 bit depth.

Hence (I guess), short path from high quality video frame to Metal texture via MTKTextureLoader is probably not for you.

What I would do instead, is to get access to real data in memory, and write Metal kernel performing manual color loading and conversion, for example from HLG_BT2020 directly into MTLPixelFormatRGB10A2Unorm.

Some 6 years passed since I worked with raw video and Metal, but I seem to remember that you can go CGImage -> CVPixelBuffer -> raw data pointer route to have something that can be read by Metal kernel (or shader for that matter) and then one can write or render to texture the usual way.

0