Creating Metal Textures from kCVPixelFormatType_Lossless_420YpCbCr10PackedBiPlanarVideoRange ('&xv0') buffers

I'm testing on an iPhone 12 Pro, running iOS 17.5.1.

Playing an HDR video with AVPlayer without explicitly specifying a pixel format (but specifying Metal Compatibility as below) gives buffers with the pixel format kCVPixelFormatType_Lossless_420YpCbCr10PackedBiPlanarVideoRange (&xv0).

_videoOutput = [[AVPlayerItemVideoOutput alloc] initWithPixelBufferAttributes:@{ (NSString*)kCVPixelBufferMetalCompatibilityKey: @(YES)
}

I can't find an appropriate metal format to use for these buffers to access the data in a shader. Using MTLPixelFormatR16Unorm for the Y plane and MTLPixelFormatRG16Unorm for UV plane causes GPU command buffer aborts.

My suspicion is that this compressed format isn't actually metal compatible due to the lack of padding bytes between pixels. Explicitly selecting kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange (which uses 16 bits per pixel) for the AVPlayerItemVideoOutput works, but I'd ideally like to use the compressed formats if possible for the bandwidth savings.

With SDR video, the pixel format is the lossless 8-bit one, and there are no problems binding those buffers to metal textures.

I'm just looking for confirmation there's currently no appropriate metal format for binding the packed 10-bit planes. And if that's the case, is it a bug that AVPlayerVideoOutput uses this format despite requesting Metal compatibility?

Answered by tangobravo in 803714022

It looks like ShinyQuagsire has discovered the answer here: https://developer.apple.com/forums/thread/758804

There are some private Metal Texture formats accessible from AppleGPUFamily4 that also do RGB conversion. So I guess the fact the these do exist means technically that format is "metal compatible" but it would be great to have them declared in public headers and documented so they can be used by third party apps.

Accepted Answer

It looks like ShinyQuagsire has discovered the answer here: https://developer.apple.com/forums/thread/758804

There are some private Metal Texture formats accessible from AppleGPUFamily4 that also do RGB conversion. So I guess the fact the these do exist means technically that format is "metal compatible" but it would be great to have them declared in public headers and documented so they can be used by third party apps.

Creating Metal Textures from kCVPixelFormatType_Lossless_420YpCbCr10PackedBiPlanarVideoRange ('&xv0') buffers
 
 
Q