ARKit video YCbCr -> RGB conversion matrix

Question

Hi all,

though I have the YCbCr texture of the camera feed, I still lack the right matrix to convert it to what ARKit is showing as background texture.

I tried it with this matrix (found here):

constant const float4x4 ycbcrToRGBTransform = float4x4(
    float4(+1.0000f, +1.0000f, +1.0000f, +0.0000f),
    float4(+0.0000f, -0.3441f, +1.7720f, +0.0000f),
    float4(+1.4020f, -0.7141f, +0.0000f, +0.0000f),
    float4(-0.7010f, +0.5291f, -0.8860f, +1.0000f)
);

But this one is too bright.

Are there docs about this or is the background_video_frag source available? (Xcode tells me it's not/needs to be loaded, but I don't know from where)

I tried to find out from the buffers used in background_video_frag, but without the source it's hard to tell what's what.

495

Posted by

AlgoChris

Reply

Hi AlgoChris,

Your color conversion matrix looks correct. Is is possible that the background texture (drawable) is using MTLPixelFormatBGRA8Unorm_sRGB? If so, can you try changing it to MTLPixelFormatBGRA8Unorm? The fact that your resulting images is too bright is most likely be caused by linear->sRGB conversion being applied twice.

—
Graphics and Games Engineer
Ok, seems my answer didn't fix this problem at all. It is still too bright.

Which texture do you refer to when you talk about the background texture? Color 0? (I use SceneKit/SCNProgram to get my stuff on screen)

—
AlgoChris
Ok, I think my question should be:

How do I change the color space in a SwiftUI app (IOS)?

—
AlgoChris

Answer 1

Spot on, thanks a lot :)

import ARKit

public class MyARScnView : ARSCNView {
    public override var colorPixelFormat: MTLPixelFormat {
        get {
            return MTLPixelFormat.bgra8Unorm
        }
    }
}

fixed my issue

Posted by

AlgoChris

Add a Comment

Answer 2

Spot on, thanks a lot :)

import ARKit

public class MyARScnView : ARSCNView {
    public override var colorPixelFormat: MTLPixelFormat {
        get {
            return MTLPixelFormat.bgra8Unorm
        }
    }
}

fixed my issue

Posted by

AlgoChris

Add a Comment

ARKit video YCbCr -> RGB conversion matrix

Accepted Reply

Replies